can anyone check if i've set up my robots.txt file properly? i'm not entirely sure how to do it but i do NOT want AI feasting on this site https://lapistoria.neocities.org/robots.txt
1 like
starry-knight
10 months ago
Yeah I think you did it right! That's pretty much how mine looks except I allowed Googlebot and DuckDuckbot so my site is indexed by search engines. I would also recommend making a sitemap.xml file if you allow those bots. It allows them to find and index your web pages more easily :3
1 like
universial truth of bolt: i love my friends