I’ve been writing an open source email (SMTP,IMAP and POP) server in Erlang called ErlMail at http://erlsoft.org which has been happily taking up all of my spare time.

http://erlsoft.org is the main website for the ErlMail project as well as a DNS server, VoIP server and a web server with it’s own markup language.

All of the projects are on Google Code now as well at the following URLs:

• http://erlmail.googlecode.com - Email Server
• http://erlweb.googlecode.com - Web Server
• http://erldns.googlecode.com - DNS server
• http://erlvoip.googlecode.com - VoIP server
• http://eif.googlecode.com - Erlang Internet Framework - updates and manges the other projects
• http://erlyvideo.googlecode.com - Flash Media Server

Well, the lets you know where I’m at and what I’ve been up to for the past few months.

The sitemap protocol has grown beyond just Google these days. You can find out more information about the protocol itself at sitemaps.org. They give a good overview of the protocol, which is pretty simple, and in the FAQ they imply that other search engines are using the sitemaps protocol.

That fact turns out to be true. For some time now Yahoo has been using the sitemaps protocol as well. On their Site Explorer web site you can let them know about your sitemaps in much the same way that you would inform Google and they also take feeds, such as RSS feeds, in addition to the sitemap protocol.

MSN is also suppose to be using the sitemap protocol, but I haven’t found the page to submit any sitemaps to.

At this point the only thing that I can think of that would make this better is if there was a way to for auto discovery of sitemaps. Something similar to the auto discovery options for RSS feeds, a META tag that point to one or more sitemaps on your domain. 

With this type of auto discovery method any search engine could use the sitemap protocol and have an intelligent way to spider your site without having to login to all of the different services that use sitemaps. 

sitemaps, Google, Yahoo

Technorati Tags: sitemaps, Google, Yahoo

I’ve been moving my servers around and I am no longer hosting the servers at my home. I eventually plan to host these servers on my own hardware sometime off in the future, but my own personal plans making hosting my own server nearly impossible at the moment.

So I’ve recreated the site as best I can on a WordPress blogs. I’m still reworking some of the content to make sure the URL are as similar as possible, but some of the content will be lost in the transition.

If you notice something that you want back, email me and I’ll re-post the content in blog form instead of articles or whatever format they were in.

Spider Hunter

Technorati Tags: Spider Hunter

SpidErl will mostly just fetch things in the first few versions, I’m going to be collecting RSS feeds and some Amazon affiliate data for the most part. I think I’m going to write a module that collects AdSense data as well, obviously you need a user name and password to get that data. In the long run I hope to create a full fledged Search Engine around SpidErl’s information store, but that will be month or years from now.

In the mean time look for some notes about building my own spider and a site redesign taking advantage of the information that the spider collects

What in fact is happening is that Googlebot is able to see the contents of the paid area for the New York Times without actually paying. This is shown by the fact that when someone searches for content on the new York times web site the information displayed in the Google pages is different than what a non subscriber would get if they went to the New York Times website.

This has always been one of the arguments I have seen in favor of cloaking. The ability to have your content indexed by certain IP addresses for search engines spiders, while not allowing people to see the content without at least some sort of registration. In my mind this is 100% in cloaking technology, but this is the type of cloaking that should be allowed. Keeping that in mind, the Google terms of service does not allow cloaking of any kind. So while I completely support this use of cloaking technology it seems only fair that Google would have to ban all of the web pages from the New York Times website. This is an obvious violation of terms of service and if Google makes an exception for a predominant web site such as the New York Times they leave themselves open two potential legal actions from site owners who have been banned under this policy.

Personally I don’t like either of those options. My personal preference would be to see the Google terms of service modified in such a way that cloaking technologies were allowed so long as certain criteria are met, with the intention behind those criteria to be displaying the same or extremely similar content to both the search engines spider and the end user. Of course that opens up another completely different set of problems where Google would end up having to view the contents of both a cloaked and uncloaked pages and then make their decision on whether or not the cloaking technology was the appropriately. That gets us into a whole subjective realm of what constitutes similar content.

So the bottom line here is that it would appear that Google is damned if they do anything and if they don’t. The chances of getting the New York Times to stop using the cloaking technology would appear to be slim at best and now that this is public knowledge if Google does nothing they may see other consequences.

http://blog.outer-court.com/archive/2006-06-19-n18.html

About a month ago she made a post in which she shows that her log entries verified that the Google AdSense media partner GoogleBot was adding information into the Google search engine. While Google maintains that they have different search engines spiders for different purposes, it certainly does not surprise me that the information is being used across their different services.

In fact it would be the best use of their resources to have a single search engine spider that collected all information about a website and then the each of their different services used that one huge database differently. I do understand Google not wanting people think that having an AdSense account would get them into the search engine results there are faster in any way. It just seems to me to be a huge waste of resources.

http://www.jensense.com/archives/2006/04/adsense_mediapa.html

The Google site map program seems to be obsessed with proving that you own the page that you’re tracking. Which all in all is not a bad thing, but initially they do not have enough ways to prove you’re the owner of the site. While they have been adding more ways to validate your site in the past few months the initial way of validation was a file that could be put in the root of your home page.

In my case the site map service would not validate the file because my 404 error messages were not always returning 404 error codes. This was done on purpose and was something that I was unwilling to change.

Interestingly enough I had several of my sites that validated just fine. All the sites were on the same server and all the sites are configured in the exact same way, but for some reason about 25% validated fine while the rest were denied for not returning 404 error codes.

I have not tried the new META tag for site map validation yet as I am currently redesigning the core of many of my sites, but during this redesign I will be implementing the new META tag validation and I look forward to many other new features from site maps as well.

http://sitemaps.blogspot.com/2006/05/more-about-meta-tag-verification.html

This is free open source code. No need for any licensing at all. There is no warranty and I will not be providing any technical support.

I highly recommend that you use the code as a basis for your own implementation and not use this code directly. Especially in a production environment.

I know of some issues that i has regarding how it does log-ins and I had always intended to somehow cache the log-in so that it would be able to do multiple transactions at once. If you find the code helpful, please put up some comments about it and if you feel like re-releasing the code I’d love to hear about it.

Download JavaMail IMAP CFC

And for the record, the comment sections have code that tells spider to not follow any of the links and the Google AdSense code to not use the comments as part of the page they use to target the ads. So from a spammers perspective, putting up your link on this site will not help you in any way …