daikonet
1 year ago
u can opt out though! you need to create a robots.txt file inside ur site folder and then copy and paste the contents of this pastebin inside and remove all the # symbols in the list to stop said crawlers from whipping shitties all over ur webzone: https://raw.githubusercontent.com/neocities/neocities/refs/heads/master/views/templates/robots.txt
mr-pibbs-site
1 year ago
i'm slightly concerned that if another crawler comes out that's not on this list i'll have to update it manually, but if i just do the User-agent: * and Disallow: \ that should work on everything right? not concerned with showing up on search engines personally
seashrine
1 year ago
i have a robots.txt but do the #'s need to be removed to work? just to double check on some things
daikonet
1 year ago
@mr-pibbs-site um! i believe if you remove the # at the very bottom with the "Disallow: /" it should block all crawling, yes. you may wanna remove the "Allow: /" line near the top as well!!
daikonet
1 year ago
@seashrine yes, the # symbols are commenting out those lines! removing them will make the robots.txt block the bot on that line (i removed all of them, but left "Disallow: /" with its # so my website will still show up on google searches)
seashrine
1 year ago
@daikonet thank you! i saw a different one going around and wasn't sure if they were to be removed or not, thanks for clearing that up!
daikonet
1 year ago
update i did it and one of my drawings got blasted with downvotes.............. i undastand........
mouette
1 year ago
thank you so much daiko!!! i've done likewise :D super humbled to be buttoned, gotta work harder on my site! ;_;
sunfishdreamworld
1 year ago
i added your button under my favorite sites section but i'm not sure if that has anything to do with it ^^;
OMG!! thank you!!