Spider Hunter

I was just going through my notes on tracking search engine spiders and I thought I’d add in some robots.txt data into the IP database Turns out that in the past 20 months only 4,532 distinct IP addresses have ever looked at a robots.txt on my system. Interesting enough, but it sure does cut down […]

Last night I decided to put the new IP database to the test to find out the data that I was looking for in the end. I am moslty interested in tracking search engine spiders, so here is what I found. Starting with the total IP addresses in the database, 568,286, I then wanted to […]

I just upgrade the MySQL database that runs all of my sites and changed a few options to try to get some more speed and a few new feature, well not new features as much as bugs fixed in a feature I was trying to use. MySQL has a way that you can insert data […]

I spent all day today working on getting some RBL data into the IP database. I am now collecting RBLs on 27,000 IP addresses that I have confirmed have sent me email messages. The funny thing is that when you are looking at the IP addresses alone, more then 60% of the IP addresses that […]

I just setup the cron job to run the databae updates at 1am and 1pm every day. This will update the visits, cookies, referers, and the IP to country data, along with doing some database clean up. We’re getting close to having a real live working IP database here

I just added the IP to country database feaure to the IP database at http://ipd.spiderhunter.com It takes the data from all of the IP address registrars and compiles it into one database that has the start IP address, the end IP address and the CIDR number. I had posted a link a few days ago […]

I’ve posted similar things to this before on different sites that I run, but it keeps becoming more and more true the more I know about computers. I’ve been working on the IP database today, mostly taking my prototype scripts and rewriting them into languages that were better suited. I started out with ColdFusion, as […]

A few days ago I found a MailToaster that looks like will be able to handle nearly everything I want an email server to do: anti-spam, anti-virus, virtual domains and much more. It’s all based on open source software and promises to be a great resource for anyone. I’ll be talking about it more at […]

I was reading over at Webmaster World and I found this gem: http://www.webmasterworld.com/forum5/6053.htm The basic idea it that you can now describe certain links as link that you do not want to give credit to on your page rank. This is being designed for Bloggers, but I think this is going to have an effect […]

I woke up this morning to a server that wouldn’t boot, it wouldn’t even make a noise. Turn out to be half of the RAM died in the server at about 2am. Took me nearly an hour to get the server back up and running, so I’m sorry if anyone tried to visit and couldn’t […]