29 Jan
Robots.txt info in IP Database
I was just going through my notes on tracking search engine spiders and I thought I’d add in some robots.txt data into the IP database Turns out that in the past 20 months only 4,532 distinct IP addresses have ever looked at a robots.txt on my system. Interesting enough, but it sure does cut down on the IPs to look at when looking for search engine spiders. In any case the robots.txt infomraiotn will be part of the subscription service at http://ipd.spiderhunter.com
