Every once in a while I get these massive view spikes and I'm not sure if it's one of those facebook nostalgia groups highlighting me without telling me or if I'm being scrapped by AI bots.
10 likes
Delete
Are you sure you want to remove this comment from the news feed?
it doesn't look like you have a robots.txt file in your index directory. that'll take care of a lot of the AI bots (not all of them, but most of them). you can use mine as a reference if you like.
since the homepage of every website is in the index (or root) directory, you can add "/robots.txt" after anyone's root URL and it will either display one or it'll return 404. you can't view the entire directory this way, but you can see if someone's got a file of a particular name.
You have a wonderful collection of writing and fun, evocative pictures. But! The fact that you have a sort of "screen saver" made me make a loud noise (In a good way!)
2 likes
Delete
Are you sure you want to remove this comment from the news feed?
it doesn't look like you have a robots.txt file in your index directory. that'll take care of a lot of the AI bots (not all of them, but most of them). you can use mine as a reference if you like.
how do you see an index directory on purpose, (aside from site editor) i only get that when i half break a link
since the homepage of every website is in the index (or root) directory, you can add "/robots.txt" after anyone's root URL and it will either display one or it'll return 404. you can't view the entire directory this way, but you can see if someone's got a file of a particular name.
In general I still want my site to be scraped for search engines but my settings disallow scraping for AI: https://wren.zone/robots.txt