f.weber/homebrewery
0
0
Fork 0
mirror of https://github.com/naturalcrit/homebrewery.git synced 2026-11-04 22:38:43 +00:00
Code Wiki Activity

Merge pull request #5074 from ericscheid/exclude-AI-scraper-bots-5073

Browse Source 
disallow clanker access to user content
This commit is contained in:
Trevor Buckner
2026-09-30 13:37:10 -04:00
committed by GitHub
parent 74ba580296 658eab7ee7
commit c636cce7af
1 changed files with 14 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
robots.txt
+14
View File
@@ -2,3 +2,17 @@
User-agent: * User-agent: *
Disallow: /edit/ Disallow: /edit/
# AI / LLM training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
Disallow: /share/
Disallow: /download/
Disallow: /print/
Disallow: /edit/