Neocities.org

Mike Grindle's Webpage

mikegrindle.com

428,357 views
70 followers
2,213 updates
0 tips
Here’s a friendly reminder to all you wonderful web masters out there to add/update a robots .txt file to your site if you don’t fancy bots scraping everything you’ve made.
7 likes
mikegrindle 1 month ago

A new enabler of mediocrity to add to your file: Applebot-Extended

1 like
5 likes
getoffmylawn 1 month ago

Thank you, keep forgetting to create one and just did it!

2 likes
owlroost 1 month ago

One to add for you: User-agent: TurnitinBot

3 likes
colexdev 1 month ago

Thanks! I have one that is pretty extensive so far, I will add anything from yours that I do not have as well as the TurnitinBot that @owlroost mentioned! Thanks all

2 likes
punk-tart 1 month ago

Do we just add it anywhere on our website?

2 likes
mikegrindle 1 month ago

Thanks for the recommendation, Owlroost. I’ll be sure to add that. @punk-tart, it should go in your site’ root directory (so, not in a “folder”, but the same place as you homepage - hope that makes sense). Cheers all.

2 likes
punk-tart 1 month ago

Thanks for the clarification, added and good to go! Always appreciate dodging AI

2 likes

Website Stats

Last updated 12 hours ago
CreatedNov 3, 2022
Site Traffic Stats

Tags

writing essays blogging technology links