RSL Progress
I was in Yosemite all weekend and during the drive I was thinking about how I would like to implement the Real-time Spider List. First off I found that TinyDns from D. J. Bernstein at http://cr.yp.to/djbdns.html . This is the same guy who started Qmail as well, at least that is the way I remember it 
In any case good, quick, small, easy to use software. Now that we have the software down, I was thinking about the list itself. I’m actually thinking of having two or three separate lists. First off will be the spider checker. This will respond with the information about what search engine a search engine spider is associated with, give an unknown answer or give no answer at all. The second list will be a quality checker. Something that the members here at spiderhunter.com will be able to influence. The member will be able to state that the spider is real and good or not. Then the quality checker will give a response from 1 to 100. This being a percentage 100 will be the best and 1 will be the worst. Lastly I was thinking of a visit checker. Something that will respond with how many days since a spider has been seen on one of my domains. I may offer to have data from other domains in the future as well, but to start my 50 or so domains will have to do. Anyway, this will give a number between 1 and 254 as to how many days it has been since the spider visited. A response of 254 will mean that the spider has not been on one of my domains in 254 or more days. If anyone has any more ideas on this please email me or post over at the forums.
