Spider Hunter

When I was going through the IP to country script to create the IP database I noticed a number in the data files that seemed to have nothing to do with IP addresses. This was an Autonomous System Number, or ASN. An ASN is used by the routing protocol called BGP. I don’t understand the […]

I was wandering and reading on the Internet this morning and I found a reference to a certain ISP that used Three digit airport codes in thier reverse DNS names. This of course made me start to think about other ISPs, that may be doing the same thing. If nothing else this would help get […]

I’ve been spending time today working on the IP database and the members section for the Web site. I finalized some of the members stuff at http://members.spiderhunter.com Most notably I have the IP to Country database downloadable to anyone who has an account. You can get it from http://members.spiderhunter.com and then going to the members […]

I was just going through my notes on tracking search engine spiders and I thought I’d add in some robots.txt data into the IP database Turns out that in the past 20 months only 4,532 distinct IP addresses have ever looked at a robots.txt on my system. Interesting enough, but it sure does cut down […]

Last night I decided to put the new IP database to the test to find out the data that I was looking for in the end. I am moslty interested in tracking search engine spiders, so here is what I found. Starting with the total IP addresses in the database, 568,286, I then wanted to […]

I just setup the cron job to run the databae updates at 1am and 1pm every day. This will update the visits, cookies, referers, and the IP to country data, along with doing some database clean up. We’re getting close to having a real live working IP database here

I just added the IP to country database feaure to the IP database at http://ipd.spiderhunter.com It takes the data from all of the IP address registrars and compiles it into one database that has the start IP address, the end IP address and the CIDR number. I had posted a link a few days ago […]

I’ve posted similar things to this before on different sites that I run, but it keeps becoming more and more true the more I know about computers. I’ve been working on the IP database today, mostly taking my prototype scripts and rewriting them into languages that were better suited. I started out with ColdFusion, as […]

I have been looking for a way for quite some time to figure out the CIDR numbers for an IP Address. Until today I was thinking I was going to have to data mine the IP registrars, but the IP to Country database in the an earlier blog is giving me a way to do […]

I found a great link earlier today, I was researching how to make or use an IP to Country database and I hit the mother load at http://www.pierzchala.com/visitor-tracking.html He even gives you the PERL script to download and update the files when ever you want. I’m going to be incorporating this into the IP Database […]