16 Mar
What spiders do you care about?
Over the years I’ve gone back and forth over which spiders to track more diligently then others and which ones I simply don’t care about any more. For the most part these days the only spider I care about is GoogleBot.
I think that MSNBot is of some consequent, as is Inktomi/Yahoo’s Slurp, but Google is the search engine that I get the most traffic from, so their spider is the one I care about the most.
Of all the other spiders out there they seem to take up more of my bandwidth then they give me in kind of return.
Perhaps I’ve gotten tired of the game or is it that they game isn’t as important anymore? Knowing that spiders do visit your site is good information, but do you track how often and which page they go to? How detailed do you get?
