does anybody know how to get yourself off grokipedia.... or stop grokipedia from scraping your website.... my stupid yaoi rambles have gotten me sited as a source on there and i never want that to happen again.......
You'll want to add a robots.txt file to your site directory that blocks all bots and/or add something like "" to the head of your pages. No clue how to get your stuff off the site though. Maybe you could request removal but I really doubt they'd listen. Sorry that happened though that's horrible
not sure how to get it taken off but i recently made a new neocities for a project & saw that when you make a new site now, nc auto-includes a txt file preventing scraping. i put it on my site fwiw - maybe add it to yours? https://fairygore.neocities.org/robots.txt
![[tfpXE] avatar](/site_screenshots/15/25/tfpxe/index.html.50x50.webp)
You'll want to add a robots.txt file to your site directory that blocks all bots and/or add something like "" to the head of your pages. No clue how to get your stuff off the site though. Maybe you could request removal but I really doubt they'd listen. Sorry that happened though that's horrible
oh neocities cleaned the meta tag I wrote uh this site has them https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
not sure how to get it taken off but i recently made a new neocities for a project & saw that when you make a new site now, nc auto-includes a txt file preventing scraping. i put it on my site fwiw - maybe add it to yours? https://fairygore.neocities.org/robots.txt
you can remove stuff from robots.txt that already comes with the website
thanks guys! i already have a robots.txt file but i guess i had missed a couple bots to block rip