appolinaire
1 week ago
I'm not tech savvy on that level, mine wasn't correct but I don't know which crawls to block or allow
divineintrusion
5 days ago
I'd actually reccomend checking out this page in specific: https://baccyflap.com/res/robots/. They have a great list for a robots.txt file, along with a few other measures to assist in bot proofing a site!
Thanks that means so much to me :)