Neocities.org
The robots.txt situation is kinda weird for me bc yeah. Lots of ppl don't even know what it does, at the same time the fact that it allows it by default (which is web standard but whatever) is iffy. IDK I feel like there's a big disconnect between Neocities and professional web developers because of AI and it's kinda sad
2 likes
inb4 someone complains about Subway Kyle: every fucking professional web developer uses AI to some extent, it's literally industry standard. Kinda sad that that's the case lmao
https://file.garden/ZXGTjTtcLzaPWNqI/robots.txt default robots.txt for those wondering
and it's just a suggestion too, nobody -has- to honor it and a lot of AI scrapers simply don't bother as far as i've seen... hell, it's been my eternal disdain for archive.org, also ignoring robots.txt and forcing me to email them every time i get a new account because i don't want my stuff archived...
@silverheart as much as I love IA I will say that ignoring robots.txt is pretty uncool of them.