Spider Hunter

18 Aug

Current Status

1,363 known spiders from MSN, Google, Inkomi and Teoma are in the data base right now. Not bad for a weeks worth of work :-) Actually a week wort of scripting, 30 minutes of running the script. This list comprises every spider that I can validate with hits on one of my web servers. I’m 100% sure there are more, but these are the ones I can prove. I’m still working on the article, between a little bit of down time and trying to make it where someone without a CS degree can understand it, well, it’s taking longer then I wanted it too … But it’s coming. I’m actually trying to think of more scripts and reports that I need to write. I’m going to be adding some searching stuff to be able to search through the spider database. That all might end up over at www.realtimespiderlist.com in the members section, but parts of it will be here as well.

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • del.icio.us
  • Netvouz
  • DZone
  • ThisNext
  • MisterWong
  • Wists

Leave a Reply

You must be logged in to post a comment.

© 2009 Spider Hunter | Entries (RSS) and Comments (RSS)