f.weber/homebrewery
0
0
Fork 0
mirror of https://github.com/naturalcrit/homebrewery.git synced 2026-11-04 20:29:01 +00:00
Code Wiki Activity
10,303 Commits 72 Branches 40 Tags
f2e7ec59dee03c92fef731a7aaeec4ac092bfc41
Commit Graph
5 Commits
Author SHA1 Message Date
Trevor Buckner 658eab7ee7 Combine user agents since they all use the same rules 2026-09-30 13:36:48 -04:00
Eric a74d9d9ca8 disallow clanker access to /share, /download, /print 
does not block search engine bots, just the scrapers that feed the LLM training
 2026-10-01 01:06:09 +10:00
G.Ambatte ecae16b5d4 Update robots.txt (#1239) 2021-02-19 17:54:00 -05:00
Trevor Buckner 587ce78f4e Update robots.txt 
This was blocking Google from updating the `noindex` tag.
 2020-04-21 22:46:42 -04:00
Trevor Buckner e88253f364 Added robots.txt, nofollow in metatags 
Also noindex on /print/ pages
 2020-04-20 13:46:18 -04:00