virtually-isolated.neocities.org
badgraph1csghost
12 months ago
it doesn't look like you have a robots.txt file in your index directory. that'll take care of a lot of the AI bots (not all of them, but most of them). you can use mine as a reference if you like.
geouniversal
12 months ago
how do you see an index directory on purpose, (aside from site editor) i only get that when i half break a link
badgraph1csghost
12 months ago
since the homepage of every website is in the index (or root) directory, you can add "/robots.txt" after anyone's root URL and it will either display one or it'll return 404. you can't view the entire directory this way, but you can see if someone's got a file of a particular name.
koyo
12 months ago
In general I still want my site to be scraped for search engines but my settings disallow scraping for AI: https://wren.zone/robots.txt
foxbugforest
1 year ago
Thank you so much! That means a lot, I appreciate your kind words and am glad you enjoy my site and art. ^w^
Thank you for having a look at my little yellow pages! That zine was pretty local and I gave most away by leaving them around the city. I hope it still gives a good enough idea on where to start anywhere though!
it's definitely given me a lot to think about!