f.weber/homebrewery
0
0
Fork 0
mirror of https://github.com/naturalcrit/homebrewery.git synced 2026-11-04 22:38:43 +00:00
Code Wiki Activity

disallow clanker access to /share, /download, /print

Browse Source 
does not block search engine bots, just the scrapers that feed the LLM training
This commit is contained in:
Eric
2026-10-01 01:06:09 +10:00
parent db5dadab6e
commit a74d9d9ca8
1 changed files with 42 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
robots.txt
+42
View File
@@ -2,3 +2,45 @@
User-agent: *
Disallow: /edit/
# AI / LLM training crawlers
User-agent: GPTBot
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: ClaudeBot
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: CCBot
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Google-Extended
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Bytespider
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Applebot-Extended
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Meta-ExternalAgent
Disallow: /share/
Disallow: /download/
Disallow: /print/
User-agent: Amazonbot
Disallow: /share/
Disallow: /download/
Disallow: /print/