Spider Hunter

I’ve been playing with Google’s new Sitemap service today.

Seems that Google is looking for a way to let Webmasters tell them exact what pages are on their web sites and when they are updated. It is a good idea for Google as they can have webmaster give them hints on how often to spider their sites and which pages are more important.

Of course this will be abused, but Google is pretty clear that they are only going to use the site map service to learn better about the content of web sites, they are not going to use it as hard and fast data to throw into the index.

And the people who are going to get accused of abusing it are going to be the early adopters that fully integrate it into their sites. So long as it doesn’t directly effect the search ranking, there really isn’t a profit motive for abuse …

I for one am grateful to have a way, that I have already automated into many parts of my system, to tell GoogleBot about obscure pages. Sometimes there is content on some of my sites that I only link to one or very few times. The chances of those pages getting spiders are much smaller then the pages that I link to from every page on my site. This will point the spiders in the right direction and it won’t effect my page rank.

Although now that I think about this, it will end up increasing my page rank as those obscure pages will only link to things inside my own site. This is just getting better and better ….