I mis-id'd scrubtheweb's user agent. It was called Scrubby, not roverbot. Corrected and included a link to an archived version of the site where I found the robot listed.
Thunderstone's has some neat links. I was just browsing through several personal websites focusing on astronomy. It's really a shame these sites are practically invisible on modern search engines.
As someone who's always been fascinated with search engines it was an enjoyable read.
Thanks.
Thunderstone's has some neat links. I was just browsing through several personal websites focusing on astronomy. It's really a shame these sites are practically invisible on modern search engines.