Spider Hunter

Doh! The URL for the wiki page is http://wiki.spiderhunter.com and you can click on a link on the left bar

I added some basic Wiki pages over the weekend. The idea behind a Wiki Page (Wiki means quick in Hawaiian) is that you have a series of openly editable articles that form a knowledge base for a site. In this case pages that talk about IP Tracking, Search Engine Spiders, Search Engines and the like. […]

I just went through my IP database, which now stands at 383,472 IP addresses checked :-), looking at the IP addresses that their percentile score is 0.75. This means that the IP Address has never had any cookies. It has never had any referrer data and it has a known good spider user agent. 3 […]

I just finished testing the DNS server at realtimespiderlist.com. It’s up and running and loaded with all the data I have collected on spiders so far. I’m going to be writing some pages later today at http://www.realtimespiderlist.com to explain how to use the system.