Developing a bot list
I was just thinking about creating a list of user agents that bots are known to use and I had the thought that I might be putting the cart before the horse. To do that I would need to have a list of known bots. This of course would be a much larger list than a spider database, as bots can be used for just about anything. This list would include the non-spider that I have mentioned before and would also include the search engine spider too. This being the easiest way to get a listing of all user agents that bots use I’m going to start on it right away. The code should be pretty similar for the spider checkers, except the IP check would not be needed and a visit check might be nice. Since the primary data would be if the user has no referrers and no cookies, if there is only one visit this could be misleading. So a minimum of say 10 visits with no cookie and no referrer data would be needed.
