f.weber/homebrewery
0
0
Fork 0
mirror of https://github.com/naturalcrit/homebrewery.git synced 2026-11-04 16:08:54 +00:00
Code Wiki Activity

Combine user agents since they all use the same rules

Browse Source
This commit is contained in:
Trevor Buckner
2026-09-30 13:36:48 -04:00
committed by GitHub
parent b6b709d1a7
commit 658eab7ee7
1 changed files with 1 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
robots.txt
+1 -29
View File
@@ -5,42 +5,14 @@ Disallow: /edit/
# AI / LLM training crawlers
User-agent: GPTBot
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: ClaudeBot
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: CCBot
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Google-Extended
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Bytespider
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Applebot-Extended
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Meta-ExternalAgent
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Amazonbot
Disallow: /share/
Disallow: /download/
Disallow: /print/
Disallow: /edit/