Neocities.org
heya, i remember you posted something about robots.txt earlier -- that's mostly a convention with no guarantee anyone on the web will follow it. you can request x-bot not to crawl your site but they still can if they want to.
3 likes
i'm sure you already know! but i just wanted to leave the note! openai seems to be ignoring it for example: https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt
Yeah it is very unfortunate that they will not follow it. I know this doesn't apply to neocities, but for people that host their own sites I recently heard cloudflare released a feature to block AI bots.
Absolutely, all the robots file does is state that you do not consent - whether companies listen to that (often, they don't) is another matter. I think it's worth doing, but I didn't mean to create a false sense of security.