I mis-id'd scrubtheweb's user agent. It was called Scrubby, not roverbot. Corrected and included a link to an archived version of the site where I found the robot listed.
Added an article about ScrubTheWeb, it's history and closure.
5 likes
melonking
8 years ago
As someone who's always been fascinated with search engines it was an enjoyable read.
1 like
arkmsworld
8 years ago
Thunderstone's has some neat links. I was just browsing through several personal websites focusing on astronomy. It's really a shame these sites are practically invisible on modern search engines.
1 like
Added a few links to ScrubTheWeb article. Some at the end, others in the article itself.