caesar.wtf
isn't there a basic one that everyone uses? unless that's not 'proper'
I'm not tech savvy on that level, mine wasn't correct but I don't know which crawls to block or allow
I'd actually reccomend checking out this page in specific: https://baccyflap.com/res/robots/. They have a great list for a robots.txt file, along with a few other measures to assist in bot proofing a site!
![[tfpXE] avatar](/site_screenshots/15/25/tfpxe/index.html.50x50.webp)
![auberylis [dot] moe avatar](/site_screenshots/34/10/astrossoundhell/index.html.50x50.webp)
isn't there a basic one that everyone uses? unless that's not 'proper'
I'm not tech savvy on that level, mine wasn't correct but I don't know which crawls to block or allow
I'd actually reccomend checking out this page in specific: https://baccyflap.com/res/robots/. They have a great list for a robots.txt file, along with a few other measures to assist in bot proofing a site!