Proof that Googlebot looks at HTTP status codes
This is a question that I have pondered over a a while, if GoogleBot, or any spider for that matter, looks at the HTTP status codes.
Now we know that at least GoogleBot does.
In a recent post on the Google Site map Blogs they talked about Verifying your site- trouble with 404 pages.
Basically they are trying to get people to verify the fact that they own a particular website and the associated site map by placing a file in the root for your website. If your web server is configured to give anything other then an HTTP 404 message on pages that are not found, as many sites with customized 404 error pages are, then you cannot verify your web site.
If nothing else this proves that GoogleBot does look at HTTP status codes, which is should, but this also means that Google’s Site Map program may not work as well for people with customized 404 error pages.
Frankly I’d rather have my site verifies by email or whois rather then by putting up an empty file that clutters my website and I’m planning on leaving my custom 404 error pages exactly the way they are.
