ATTEEEEEENTION: I'd like everyone to quickly check their website's stats real quick. I checked mine just now and i saw a huge increase in the number of visits, but there were a lot less hits than their should be on normal occassions
This means that more people are visiting the site but less files are being sent. This usually is what WEB CRAWLER BOTS look like. It's surprising just how many there seem to be though, so I'm wondering if anyone else has noticed this.
![xX [ TR/BZ ] Xx avatar](/site_screenshots/41/77/toribytez/index.html.50x50.webp)
This means that more people are visiting the site but less files are being sent. This usually is what WEB CRAWLER BOTS look like. It's surprising just how many there seem to be though, so I'm wondering if anyone else has noticed this.
i had a huge spike in both hits and visits recently but it's going down now. i did update my robots.txt file just in case
I hadnt even considered robots.txt, honestly a really good idea. Is there some kind of list with (public) crawler bots which I could browse?
you can copy mine if you like. i updated it from a list on codeberg 2 months ago
That's a pretty extensive list, thank you. Though I do wonder why stuff like GPTBot isn't in the list?