Item2322: Don't send Status 200 for non-existent pages

When a user enters an invalid lik, i.e. a link to a non-existing page, Foswiki gives a form where users can search for the page or create the named page. This page is delivered with status 200.
Example: http://trunk.foswiki.org/Sandbox/Some/nonexistant/name

This practice has a number of drawbacks:

1. Violates HTTP, which requires 404 for missing documents
2. Breaks automatic link checking tools, as to them the page seems to exist
3. Makes log file analysis difficult, because one can't see which pages do exist
4. Robots might fail to detect errors and save forms instead

The issues are particularly troublesom in scenarios where Foswiki is used as a CMS, where links to non-existing pages are considered an error, where few people edit the pages but many may view them, and where the fact that this is a wiki is of little interest to the usual visitor.

I know there is at least one browser out there which in its default configuration replaced 404 pages with its own content. I would however consider this a bug on the part of the browser, and if I remember correctly in recent versions of IE this only affects "short" error pages, whereas the generated form should get treated as a "long" one.

So I'd want Foswiki to send the same page it does so far, but send it with "Status: 404" instead of the 200 it uses now. If you absolutely don't want this as the default, maybe make it configurable. You could even choose the proper reaction based on the reported user agent, i.e. send 200 to IE and 404 to everyone else. I guess all of this would be more work than simply sending a 404, though, and hope for IE users to update or reconfigure their browsers.

-- MartinVonGagern - 02 Nov 2009

Related to Item897, but that's more concerned with page content and the topic creation form in particular, while I'm more concerned with HTTP headers here.

-- MartinVonGagern - 02 Nov 2009

One other observation: with a slight change in capitalization, the above example URL gives a 403 instead of a 200:
http://trunk.foswiki.org/Sandbox/Some/Nonexistant/name
This fact, i.e. the status of a broken link depending on capitalization, feels really weird to me.

-- MartinVonGagern - 02 Nov 2009

If a 404 makes old IE show its own error page then it would create a major problem for all of us that live in corporation forced to use old IE6

So please do not fix things claiming it is OK because the old browsers are broken. They are a fact of corporate life. We must ensure that all users get the correct page to create a new topic.

A configure option is bad. It is plain wrong to cut off IE6 users and lure inexperienced admins into such a trap.

Doing it for non IE agents may fly. But do we actually have a problem worth fixing here?

I fear we can break much more than we fix

http wise the 200 is not a violation as we do return a valid html page

-- KennethLavrsen - 04 Nov 2009

I had assumed that the feature to replace short error documents but to display longer ones was introduced in IE 6, but further research indicates that even IE 5 had this. MS KB 294807 Method 2 explains that IE5 in its default configuration will display error pages to the user if they are more than 512 bytes long. I guess Foswiki search pages should easily have this size, and if in doubt, you could pad them with e.g. spaces.

This observation makes considering scenarios with short error pages of theoretical interest only. For these documents, it seems that all versions since 5 would replace the error document with their own version, but only in the default configuration. So it's not the browser version which I consider broken, but the configuration. MS KB 294807 Method 1 shows how this setting can be changed. So even if you are forced to use an outdated browser, and even if some error page were too short to be displayed in all cases, then a simple browser reconfiguratoon would do the trick.

While the returned page is a valid html page, it's not the document the user requested, so I still consider 200 to be wrong. Error documents have to be valid as well, so the format of the content doesn't say a thing about the status code to use.

-- MartinVonGagern - 04 Nov 2009

I'm going to check in a one-line fix for this in trunk. We need to test that it doesn't break older browsers.

I've compared Foswiki to Wikipedia when dealing with missing topics. Wikipedia returns 404, Foswiki 200.

-- GeorgeClark - 07 Apr 2010

Added myself to test IEs when I get time. Set to minor/1.1

-- PaulHarvey - 07 Apr 2010

i'm very glad to finally see this fixed! perhaps we're now http spec compliant? this fix should also go in a 1.0.10 maintenance release, so checking in this to he release branch.

-- WillNorris - 07 Apr 2010