Spider Hunter

I’m a big fan of Jenstar from webmaster world as well as her JenSense web site. For the most part I’m interested in the information about AdSense and other contextual advertising mediums, occasionally she surprises me with information that is usually a bit more technical than I would expect. (Not that Jen isn’t technical, but […]

This is a question that I have pondered over a a while, if GoogleBot, or any spider for that matter, looks at the HTTP status codes.
Now we know that at least GoogleBot does.
In a recent post on the Google Site map Blogs they talked about Verifying your site- trouble with 404 pages.
Basically they are trying […]

The GoogleBot Media bot is the separate spider that is used to evaluate Google AdSense content. Just recently Google added a way for you to tell the Media Bot what areas of your page are more relevant then others.
Section targeting allows you to designate certain areas that should be weighted higher or lower for the […]

I was reading over at Webmaster World recently and I came across a great thread from GoogleGuy at http://www.webmasterworld.com/forum30/29720.htm
Pretty much from the get go he starts talking about great info for anyone who watches Google and the latest update, but what really caught my eye was some info about GoogleBot.
It seems that when GoogleBot sees […]

I talked about getting TimeZone data a few days ago to figure out where IP addresses are located. I realized something while working with them, I will never see the time zone data for a visitor without JavaScript. This statement alone sounds disheartening until you realize what does not have JavaScript, and that would be […]

I’m building an IP Address database. I’m going to have a lot of basic information for free and then I’m going to turn the detailed information into a subscription service. I’m going to have IP address, reverse DNS name, DNS name Search, user agent list, user agent Search, Number of cookies used per visit, number […]

Okay, spider food is not a new concept, as many of the things I am talking about here are not new concepts. Hopefully some of these are new to the readers here or I’m adding new twists to an old idea. At least I’m hoping to get more information out for public consumption then has […]

After looking at how the data is collected by the spider checker script I’m thinking of adding a few things to it. First off I want to create a non-spider list. IP Addresses that are simply not spiders. These IP Addresses will be things like aol.com’s caching computers and altavista’s babelfish. (Is that still around?) […]

1,363 known spiders from MSN, Google, Inkomi and Teoma are in the data base right now. Not bad for a weeks worth of work Actually a week wort of scripting, 30 minutes of running the script. This list comprises every spider that I can validate with hits on one of my web servers. I’m […]

Have you ever looked through your logs and found out which IP Addresses visit your sites the most? You guessed it, most of the time they are spiders or bots of some kind. In my case it tends to be Teoma. This tends to work well for finding spiders that come from a limited number of […]