limepotato/jormungandr-bite
1
0
Fork 0
mirror of https://iceshrimp.dev/limepotato/jormungandr-bite.git synced 2024-11-22 09:57:29 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

withdrawal robots patch

Browse source
This commit is contained in:
nelle 2024-06-17 13:38:18 -06:00
parent bebe68dab1
commit e93373d72c
1 changed files with 35 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

37
packages/backend/assets/robots.txt
View file

@ -1,4 +1,37 @@
user-agent: * user-agent: *
allow: / disallow: /
# todo: sitemap # explicit disallows because some bots are assholes that need that
User-Agent: Googlebot
Disallow: /
User-Agent: Storebot-Google
Disallow: /
User-Agent: GoogleOther
Disallow: /
User-Agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-agent: Amazonbot
Disallow: /