f.weber/homebrewery
0
0
Fork 0
mirror of https://github.com/naturalcrit/homebrewery.git synced 2026-11-15 12:28:45 +00:00
Code Wiki Activity

Combine user agents since they all use the same rules

Browse Source
This commit is contained in:
Trevor Buckner
2026-09-30 13:36:48 -04:00
committed by GitHub
parent b6b709d1a7
commit 658eab7ee7
1 changed files with 1 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
robots.txt
+1 -29
View File
@@ -5,42 +5,14 @@ Disallow: /edit/
# AI / LLM training crawlers # AI / LLM training crawlers
User-agent: GPTBot User-agent: GPTBot
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: ClaudeBot User-agent: ClaudeBot
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: CCBot User-agent: CCBot
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Google-Extended User-agent: Google-Extended
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Bytespider User-agent: Bytespider
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Applebot-Extended User-agent: Applebot-Extended
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Meta-ExternalAgent User-agent: Meta-ExternalAgent
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Amazonbot User-agent: Amazonbot
Disallow: /share/ Disallow: /share/
Disallow: /download/ Disallow: /download/
Disallow: /print/ Disallow: /print/
Disallow: /edit/