spotties
1 day ago
wait what? there's something i'm failing to understand, what's the point of adding the checkbox anyway if that's what you have the robots.txt file for?
tabf5
1 day ago
I think the checkbox is also a server-level redundancy, meaning that you're adding a second layer of AI-blocking along with the robots.txt
spotties
1 day ago
hmh, that does raise some concern,, i hope however this doesn't develop into a big controversy (๑•́ -•̀)
venthefrog
1 day ago
Noticed this the other day too. Very strange that they leave it unchecked, but at least they give us the option
wwwdotdeepblueme
1 day ago
Thanks, I got mine to the setting I prefer. They leave it unchecked because little to no one will voluntarily allow their sites to be scraped by AI. I do not know why neocities does it in the first place because it seems like a waste of money to allow AI to scrape sites.
sinproexe
1 day ago
literally just noticed it today... no idea since how long it was there (???) i have a robots.txt file but im not against additional protection, i guess... odd choice to default it unchecked though i agree 🤨
anypup
1 day ago
wait so if i turn off everything on my robots.txt, but leave the box unchecked, will bots still crawl my site?
gobbolab
1 day ago
Enabling this setting will trigger neocities to return HTTP 403 to known AI crawlers. This is separate from robots.txt in that it is a hard server side block. robots.txt is a preferences file you can put on your site which will tell potential crawlers what you do don't allow. Nothing forces a crawler to respect your robots.txt file however. Of note, not all crawlers are bad, example, the Wayback machine.
win1999
1 day ago
@anypup, if you have an empty robots.txt file and the box is unchecked, you will get bot scrapers.
win1999
1 day ago
Not all web crawlers harvest data for AI training, as gobbolab mentioned, many perform more essential functions like indexing search results, archiving web history, and monitoring site security. While a robots.txt file requests that bots refrain from crawling, it relies on voluntary compliance rather than technical enforcement, meaning malicious scrapers can simply ignore it.
tabf5
1 day ago
yeah I avoid using robots.txt because I still want my website to show up in search engines, but I did tick the checkbox to block the actual AI scrapers
win1999
1 day ago
@tabf5, I’d suggest a customized robots.txt file. You can target specific bots rather than making it an all-or-nothing thing. If you check that box in neocities, it'll block bots designed for Search Engine Optimization. It won't affect pages that are already indexed, but it'll prevent future crawls.
semi-sentient-goose
1 day ago
it being off by default is yet another addition to the nekoweb list lol
tabf5
2 days ago
Oh man, I forgot that part was in the tutorial. Maybe try asking @resrc for a guide on how to do that? He wrote one that also displays a viewcounter but with a new link that doesn't require a Cloudflare login. You might be able to reuse that part with dann's code too actually
maehat
4 days ago
reminds me of the guy in my college who has been playing undertale songs on the cafeteria piano just a floor below so i always hear it lol
tabf5
1 day ago
not all that much actually but i've heard his music so much that it's enjoyable to recognize in public!
tabf5
1 week ago
WHOA! There's a test video thing on /main. Check it out if you have the time! Also updated some counts on /stickers, threw an image up on /updates, and started work on /carvings. Need to sleep before I can finish the latter lmao
music-by-margaruine
1 week ago
The conflicts between the Tabling family and the Tupipe family reflect a deep, inseparable division between wood and cardboard social classes and criticizes familial and societal expectations in modern society. Bravo, TabF5! /s
garfriend
1 week ago
the walking off the screen at the end was so perfect omfg it seems that your secret calling is puppetry
strawberry-transneu
1 week ago
YOU HAD A GIANT GREENSCREEN THE WHOLE TIME??????????????????????????? i love the little puppets theyre wonderful. as well as the new friends. i hope to see more of the Tupipes. have a great day tabf5 (also how did you make this link?)
music-by-margaruine
1 week ago
reminds me of this shitpost youtuber i used to watch https://www.youtube.com/watch?v=GEihi41HgDc ^^ example ^^
tabf5
1 week ago
Thank you everyone! Maybe I'll do something more with this after all. @strawberry-transneu Fun fact: every post you make on a Neocities feed has a direct URL. Very handy! I wonder if it could even be iframe-d
dragonaero
1 week ago
ALIVE CINEMA THEORY https://pbs.twimg.com/media/HT_684vXUAE80Ck?format=png&name=900x900
*goes to websites* *searches for Random button, fails* *finds Random in dropdown* *clicks update* *tabf5 is first listing* Ah yes, working as intended
please ignore the $20 bill i slid under the table to Penelope the Cat and Kyle Drake
ohhhh that 20 bucks??? i def didn't steal it afterwards noooo that's crazy XD
guys i found 20 bucks who wants it