Hello all.
I'm wondering something.

When a visitor goes on your site an makes an error (voluntarily or not) : what do you do?
- Error 404 page?
- Redirection to your home page?
- Something else?

What do you do? What is better for search engines?
Because Yahoo regularly test my site by trying to go on non-existant pages (SlurpConfirm), and I have already seen Googlebot testing non-existant pages on my site too...

Thanks in advance for your answers.

Just an Error 404 page. We should change that of course to make it more visitor friendly.

You should *always* serve a 404 HTTP status if the file is not found. However, the 404 page can be whatever you like. Personally, I've currently got the standard 404 page but if I were a big company, I'd create an "intelligent" 404 that suggests possible pages the user may have been looking for based on the URL that was entered.

If you don't serve a real 404 when robots visit non-existent pages (i.e. you redirect them to your home page) that could really mess with your indexing and rankings...

"If you don't serve a real 404 when robots visit non-existent pages (i.e. you redirect them to your home page) that could really mess with your indexing and rankings..."
1) >> Are you sure? Don't you think Search Engines are already able to "understand" that?

2) >> Many of my visitors come on my site after watched my URL in newspapers and magazines, they directly enter the URL in the address bar, so many of them do mistakes (for example : page.htm becomes page.html, page.hmt or paeg.htm).
A survey proves that 50%+ of visitors who encountered an error page leaves the site...

I could do something like :
if(IP == IP robot || user_agent = user-agent of robot)
{ display 404 error page; }
else { redirects to the home page; }

what do you think about that?

>> 1) Are you sure? Don't you think Search Engines are already able to "understand" that?

Search engines can't understand it. They'll simply see it as a redirect – and in the worst case scenario this could be seen as cloaking. (Whilst that's unlikely, it's really bad practise to give a 200 OK response when the file isn't found.) If you really want to redirect to your home page, make sure you serve a 404 status along with the content rather than simply redirecting to your home page.

>> 2) Many of my visitors come on my site after watched my URL in newspapers and magazines, they directly enter the URL in the address bar, so many of them do mistakes...

My advice? Always use friendly URLs that don't require offline readers to enter a long path or file name as part of the URL – i.e. http://www.example.com/sale or http://www.example.com/products or http://www.example.com/magazine would be better than they typing in something like http://www.example.com/products/new/sale.htm as the address. (You can make 302 rediects to any page that doesn't have a friendly URL for this purpose – which then also makes tracking promotions and compaigns easier too.)

Of course, you could serve different responses to Googlebot (and others) but Google don't take kindly to people serving different content to Googlebot than they do their "normal" visitors...

http://www.google.com/support/webmasters/bin/answer.py?answer=35229&query=404&topic=0&type=f

We've detected that your 404 (file not found) error page returns a status of 200 (OK) in the header.

And on another page I think I read that you really get punished for redirecting 404s to other sites.

What Tony said.

Don't do redirects. Don't (!) do user agent dependent redirects. Don't even be too smart on your 404 page – the visitor must understand it's a file not found, so if your replacement is too great the point me be lost. But if you suspect that a certain URL was wanted, you can point to it of course with a friendly link below the "not found."

Ok, thanks for your advices ;-)
I will change that quickly...

~~~~~~
Tony > "Always use friendly URLs", unfortunately, I don't chose the URL in the newspapers. They talk about my site, but I'm not always aware about it!

Coming back to this.

If you customise your 404 page, there is no problem in Firefox. But in IE there's an option 'Userfriendly HTTP error messages' (freely translated from my Dutch speaking IE) and than your customised page isn't shown at all, i.s. a IE page is shown.

What is your opinion on that?

I think it's up to IE to do that, and up to us users to go with IE... maybe someone at IE figured that it takes some seconds longer for a user to understand a 404 page when it's customized, as opposed to learning to understand a repeated UI element (that white IE page).

Philipp: thank you for the feedback!