Google likes naming their spiders and giving the same name to a whole cluster of spiders. One thing that you can do with this is look up all the IP Addresses with the same name and evaluate those IPs as well. Here is the basic process:
Find one IP Address that the reverse DNS lookup resolves […]
Why? Search Engine Optimization silly! I’m not 100% sure of the status of this right now, but most search engine spider do not index dynamically generated pages. Google’s Googlebot does, but they even admit that this is limited in the amount they will index. So what is the answer? Make your dynamic pages look like […]
I was writing a blog for another one of my sites ( http://www.spamfreeemail.com ) and I was talking about Real-time Black-hole lists. Then the thought occurred to me that I have never seen this simple implementation translated over into Search engine spider detection. Using the same techniques as the RBL a Real-time spider list could […]
© 2008 Spider Hunter | Entries (RSS) and Comments (RSS)