can anyone check if i've set up my robots.txt file properly? i'm not entirely sure how to do it but i do NOT want AI feasting on this site https://lapistoria.neocities.org/robots.txt
1 like
i have been getting some trolls on the guestbook lately so i'm requiring email to sign it for a while
Website Stats
Last updated
5 days ago
CreatedMay 16, 2022
Yeah I think you did it right! That's pretty much how mine looks except I allowed Googlebot and DuckDuckbot so my site is indexed by search engines. I would also recommend making a sitemap.xml file if you allow those bots. It allows them to find and index your web pages more easily :3
noted! thank you so much!!