Feature Proposal: Simplified require-auth Status codes (401)

This is a supporting topic discussing the details of 401 responses to REST requests that fail auth. Read ImprovedRESTSupport to get the context. -- CrawfordCurrie - 09 Feb 2012

The essence of my proposal is to modify UI.pm to recognise when a request is coming from AJAX and send back a 401 rather than a 200+login page. 401 is the appropriate response according to the HTTP standards, so long as a correctly formatted WWW-Authenticate header is sent back with the response. I have tested the following code, and it works perfectly:
Index: UI.pm
--- UI.pm   (revision 13816)
+++ UI.pm   (working copy)
@@ -353,8 +353,20 @@
         $res = $session->{response} if $session;
         $res ||= new Foswiki::Response();
-        unless ( $session->getLoginManager()->forceAuthentication() ) {
+   # Check if it's an unauthenticated request to the rest script that came from
+   # XMLHttpRequest. If so, respond with a 401.
+   if ( ! $session->inContext('authenticated') &&
+        $initialContext{rest} &&
+        ( $req->http('X-Requested-With') || '') =~ /^XMLHttpRequest$/i ) {
+       # We should never get here if we are using basicauth, because if the rest script is
+       # in {AuthScripts} the auth is forced by the webserver. So we must be using a template
+       # style auth. Chuck back a 401 and see what happens.
+            $res = new Foswiki::Response();
+            $res->header( -status => 401,
+           -WWW_Authenticate => "LoginManager=\"$Foswiki::cfg{LoginManager}\"" );
+   }
+        elsif ( ! $session->getLoginManager()->forceAuthentication() ) {
             # Login manager did not want to authenticate, perhaps because
             # we are already authenticated.
             my $exception = new Foswiki::OopsException(

Note that a 401 response is only generated when:

  1. No-one is currently logged in
  2. The rest (or restauth) script is being invoked
  3. The X-Requested-With header in the request is XMLHttpRequest (works on all the browsers I have tried)
On the client, Javascript can either prompt for a user and complete the challenge-response sequence (hard) or simply alert("You must be logged in to do this") (easier). My test code simply does this:
   error: function (jqXHR, textStatus, errorThrown) {
       if (jqXHR.status == 401)
      alert("Please log in before editing");

-- CrawfordCurrie - 31 Jan 2012

Why does this need to be specific to XMLHttpRequest? Whatever is good for AJAX clients must be good for other HTTP clients too.

I have concerns about Foswiki REST support, but it seems a shame that you have hijacked Rafael's original proposal which was quite general, while your intentions are actually to smooth over a tiny part of the rest legacy.

I'd hope Rafael's original proposal here might one day evolve into a new rest2 API; so, while I think what you're doing is valuable, I'd say it'd be less misleading if you created a separate proposal for it. Especially given that the proposal text here has remained unchanged.

-- PaulHarvey - 01 Feb 2012

if you're mucking around with headers in the request, why not reply status 200, and have a response header X-error ?

Index: UI.pm
--- UI.pm   (revision 13816)
+++ UI.pm   (working copy)
@@ -353,8 +353,20 @@
         $res = $session->{response} if $session;
         $res ||= new Foswiki::Response();
-        unless ( $session->getLoginManager()->forceAuthentication() ) {
+   # Check if it's an unauthenticated request to the rest script that came from
+   # XMLHttpRequest. If so, respond with an X-Error.
+   if ( ! $session->inContext('authenticated') &&
+        $initialContext{rest} &&
+        ( $req->http('X-Requested-With') || '') =~ /^XMLHttpRequest$/i ) {
+       # We should never get here if we are using basicauth, because if the rest script is
+       # in {AuthScripts} the auth is forced by the webserver. So we must be using a template
+       # style auth. Chuck back a 200 and tell the requester the reason in the header.
+            $res = new Foswiki::Response();
+            $res->header( -status => 200,
+           -X-Error => "LoginManager=\"$Foswiki::cfg{LoginManager}\"" );
+   }
+        elsif ( ! $session->getLoginManager()->forceAuthentication() ) {
             # Login manager did not want to authenticate, perhaps because
             # we are already authenticated.
             my $exception = new Foswiki::OopsException(

   type: 'POST',
   data: formData,
   success: function(data, textStatus, XMLHttpRequest){

it really seems weird to me that everyone wants to overload (reasonably, but against the spec...) the http response codes - there seem much more consistent ways to doit.

MichaelDaum pointed out on irc that jsonrpc is great because it has a 'standard' error mechanism.... looks to me like they've just said, there's an error object that is null if everythings ok. (and a result and id attribute..)

thats pretty much the same as writing a standard that says you will have X-error and X-id and the result is in the payload, with its format and encoding also defined in the header....

On the other hand, if we think we're better and more knowledgeable about HTTP than the standard writers, who've appeared reluctant to update the use of the response codes in this way for 20years :(......

-- SvenDowideit - 01 Feb 2012

OK, lots of flames. Firstly, Paul, what I'm proposing is part of Rafael's original proposal, which is why I didn't feel the need to change it or raise a new proposal (specifically REST calls that require auth but are not provided with it (either in the form of basic auth in the request or by virtue of a cookied login) should be 401'd."

Sven, 200 doesn't work as a status response because $.ajax calls are often hidden behind other JS (e.g. in jQuery plugins). Essentially the only way to deal with the response is if it gives an error code (though yes, you could filter every single response and look for a custom header). And everyone I have googled doing this sort of thing is using 401 and WWW-Authenticate - I guess for the same reasons I want to i.e. it's obvious. I am not overloading the 401 response code, I'm using it for what it was designed for, as per the RFCs.

More of this discussion in the mail thread at http://sourceforge.net/mailarchive/forum.php?thread_name=4B73E07D.7050205%40c-dot.co.uk&forum_name=foswiki-svn and on IRC at http://irclogs.foswiki.org/bin/irclogger_log/foswiki?date=2012-02-01,Wed&sel=71#l67 Also Tasks.Item1029, Tasks.Item9184, Tasks.Item10566,

The conclusion of both these discussions is that the requirements of REST are different to those of interactive requests and should be handled with a 401. I retain an open mind w.r.t other mechanisms for handling errors from REST, but I can't for the life of me see what's wrong with 401 + WWW-Authenticate. i do not want to open the discussion on responses to non-REST requests here; this is purely about REST.

One final point that I discussed with George last night; should this 401 apply to all scripts called with X-Requested-With=XMLHttpRequest, or just to rest?

-- CrawfordCurrie - 01 Feb 2012

I didn't mean to flame (I didn't even raise an objection - just didn't want Rafael's spec to be forgotten about later).

I am concerned with why XHR should be a special case.

Aside: we can use WWW-Authenticate for auth schemes other than Basic/Digest: that's what OAuth is all about. RFC:5849#section-1.2:
Servers MAY indicate their support for the "OAuth" auth-scheme by returning the HTTP WWW-Authenticate response header field upon client requests for protected resources. As per [RFC:2617], such a response MAY include additional HTTP WWW-Authenticate header fields:

For example:

WWW-Authenticate: OAuth realm="http://server.example.com/"

But I suspect the reason why you want XHR to be a special case ... (Edit: George tells me it's to avoid the 302 redirect to /bin/login. I think we should use an Accept: (somehting non-HTML) instead).

I feel that WWW-Authenticate is inappropriate, given that TemplateLogin is an application-level thing and currently resides almost entirely outside of the HTTP-protocol-level.

Example OAuth interaction RFC:5849#section-1.2:
     POST /initiate HTTP/1.1
     Host: photos.example.net
     Authorization: OAuth realm="Photos",

RFC:5849#section-3.2 mentions 400/401 status codes.

I believe (but could be wrong) that an 401 comes with an WWW-Authenticate giving the client an HTTP-Layer option to respond to an auth challenge, and that's not what TemplateLogin (an internal, in-band, application-level thing) is.

I have now raised a concern based on:
  • the solution makes XHR a special case. That doesn't seem right.
  • 401 implies an WWW-Authenticate which implies TemplateLogin is an HTTP-Layer thing. That doesn't seem right.
I can lift it if:
  • The solution doesn't make XHR a special case.
  • I'm wrong, WWW-Authenticate can specify any old garbage, including things outside the HTTP layer, OR
  • We use X-Headers instead of (ab)using WWW-Authenticate, OR
  • We re-implement TemplateLogin to work in the HTTP layer
Edit: Tried to make this shorter.

-- PaulHarvey - 01 Feb 2012

Although I'm not sure Drupal's use of 403 is wise, here is why they do it:
... 401 Unauthorized must be combined with HTTP authentication, which is not what Drupal is using, so this is not an option

-- PaulHarvey - 02 Feb 2012

Template login is a transaction sent to bin/login which returns 200, with a user understandable (hopefully) login form. There is no error there, no place to return 401, 403 or otherwise. The login template form is a normal ok response to a request to bin/login. You can enter a request to login directly.

The question comes with the step before that. When the user accesses the "currently forbidden" resource, should they be redirected to the bin/login using 302. Or should they "fail" with a request for further authentication. As I understand Crawfords proposal, the X-Requested-With: XMLHttpRequest header is telling foswiki that this request is not from a person, redirect is useless. The return 401 with a WWW-Authenticate mechanism understandable and processed by the javascript seems reasonable to me.

The original issue way back when, if I recall correctly, was that the bin/login script returned the 40x error along with the request to login., and did not include a WWW-Authenticate request.

-- GeorgeClark - 02 Feb 2012

I was pointed to the IRC logs, and found Crawford's resistance to an X-Header solution stems from the fact that this doesn't readily propagate an error condition through some stack of JS wrapper code around an actual XHR call.

Is this what the argument boils down to?

Later: Discussing with George... I wonder if it'd be best just to ask for a Content-Type other than HTML in the AJAX request, Eg. Accept: application/json - that would make a redirect to an HTML endpoint such as bin/login unacceptable.

There's still the appropriateness of TemplateLogin as an WWW-Authenticate method; I'm not an expert on these matters, so I can accept that I might be wrong, here.

-- PaulHarvey - 02 Feb 2012

Some foods for thought: -- PaulHarvey - 02 Feb 2012

(what Crawford didn't tell us, is that he's using thirdparty ajax code, in which he can't really test for header values on success, but he can add a global ajax error handler - which is why 200+X-Error isn't suitable)

given ISSUE-13: Handling HTTP status 401 responses / User Agent Authentication Forms tells us:

2) Browsers do not display the text/html response body of a HTTP 401 response, instead, they just pop up a modal authentication dialog (until "cancel" is pressed).

the idea of using 401 always is kiboshed. frown, sad smile

Rather than hard coding this to XHTTPRequests, so I wonder if we can use the Accepts mime-type header to avoid redirecting unless we're html-ing, as TemplateAuth does not (really) support non-html auth atm.

we could do this forall requests where auth is needed but we're in a non-html situation, with a (still valid, as the RFC attempting to rewrite history havn't passed) a WWW-Authentication: Cookie ala failed draft-broyer-http-cookie-auth.

One of the reasons i would like to use the long running extensibility of WWW-Authenticate, is that ideally, rather than the error being just a naf alert(please login) dialog, we can then add a javascript popup (not a new browser window) that actually does auth for us more securely than basic auth..

-- SvenDowideit - 02 Feb 2012

I can't find any evidence anywhere that WWW-Authenticate was intended to be reserved for HTTP authentication (Basic/Digest). WWW-Authenticate just happens to be the mechanism used, and is clearly qualified to indicate the auth scheme i.e. WWW-Authenticate: Basic and WWW-Authenticate: Digest. HTTP is an application-layer protocol, but it's easy to be seduced into thinking that because the browser typically handles Basic and Digest auth that it's below the application layer. It isn't; other auth schemes already use it the way I'm proposing (e.g WWW-Authenticate: OAuth is implemented in Javascript at http://oauth.googlecode.com/svn/code/javascript/).

There is an IETF draft standard (http://tools.ietf.org/html/draft-oiwa-http-auth-extension-00) addressing the requirement for forms-based authentication. I just spent an hour reading it quite carefully.

"Servers MAY send HTTP successful responses (response code 200, 206 and others) containing the Optional-WWW-Authenticate header as a replacement of a 401 response when it is an authentication-initializing response. The Optional-WWW-Authenticate header MUST NOT be contained in 401 responses." (http://tools.ietf.org/html/draft-oiwa-http-auth-extension-00).

Of course reading IETF proposals is fraught with risk. It also says:

"If a response is either an intermediate or a negative response to a client's authentication attempt, the server MUST respond with a 401 status response with a WWW-Authenticate header instead. Failure to comply this rule will make client not able to distinguish authentication successes and failures."

This is clearly at odds with what TemplateLogin currently does (it responds with a 200, as per our prior discussions on the topic) and it will not work with browsers in their current implementations unless a non-Basic/Digest auth scheme is specified i.e. we are back to my proposed handling of a 401.

Where does this draft standard take us? Well, It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress.". However I can't find anything better.

  • HTTP is an application level protocol, and there is no reason why we should not participate in it's implementation.
  • The "basic and digest auth only" status of 401 and WWW-Authenticate is apocryphal at best. Other forms-based auth schemes use 401 and WWW-Authenticate
  • There exists a draft IETF standard for forms-based authentication.
  • The proposed handling of 401 does not conflict with this draft, though TemplateLogin as it is current implemented does.
-- CrawfordCurrie - 02 Feb 2012

  • Regarding 'application' vs 'HTTP' protocol levels, I am worried about separation of concerns. And by that I mean, I've been working under the assumption that anything calling itself an WWW-Authenticate mechanism must do all its work outside of the content body. If authenticating means picking apart some arbitrary HTML that can look like almost anything, then processing and understanding all this stuff in the response body, that's unwarranted mixing of the 'application' and 'HTTP-protocol-protocol' levels. Re-combing previously separated concerns seems incorrect. Maybe I'm wrong.
  • True, but...
  • ... I guess I need to accept that a WWW-Authenticate mechanism can expect the client to grok HTML & deal with strikeone.
  • ... and perhaps I can accept that, but you still have XHR as a special case, and I am convinced we can fix that.
-- PaulHarvey - 02 Feb 2012

After a lengthy IRC discussion, I have removed my concern, with an understanding that in addition to X-Requested-With: XHR we work towards also supporting a more friendly hint for non-browser UAs.

And at least philosophically, we agree that TemplateLogin's current strikeone implementation needs to be moved out of the HTML and into the Headers to make it more friendly to non-browser UAs, but that shouldn't block this work.

-- PaulHarvey - 02 Feb 2012

Another issue not covered yet is character encoding: while all normal requests are encoded using the site charset, it does not so on REST. These are always utf8 encoded. The current foswiki response code does not distinguish xhr calls from normal ones and does not take care of properly converting all parameters to an encoding expected by the rest of the core and plugins. This issue is covered in Tasks.Item10825 and Tasks.Item10995 where RESTish calls to save will end up in an encoding mess. So most REST applications either only just work by accident or have their own bloody workarounds, i.e. not using foswiki's own REST infrastructure at all, like it is the case for JsonRpcContrib.

-- MichaelDaum - 02 Feb 2012

... on the (de)coding front, TODO: double-check Foswiki is actually acting decoding requests according to that request's Content-Type...charset, if present

-- PaulHarvey - 02 Feb 2012

I think we all just agreed to adopt HTTP 401 for all auth, not just REST - and in the process to remove the redirect to the /bin/login script.

Crawford, please extract this feature-ette from this REST specific one - and you can consider it accepted, as its really a continuation of the 400 status task we worked on last year.

-- SvenDowideit - 09 Feb 2012

I'm confused, I don't see how this conclusion is reached. We are doing away with bin/login and template auth? I think I need to raise an objection before the 14 days elapses.

The "feature proposal" part of this still talks just about replying with 401 to XHR type requests. How does it make the jump to agreeing "to adopt HTTP 401 for all _auth, not just REST - and in the process to remove the redirect to the /bin/login script"

The move toward an improved js based auth mechanism makes a lot of sense. I have a proposal - ImproveSecurityOfTemplateAuth - that proposes just such an implementation.

Raising an objection. I don't object to the initial proposal. I object to Sven's conclusion to extend the proposal to eliminate 302 redirect for all auth. The 14 days is not up.

-- GeorgeClark - 11 Feb 2012

I hope that Crawford is willing to change all core 302's to use a direct 401 - but that commitment would need to come from him.

wrt the more important question - we talked it over on IRC, and did some testing, and then I talked to Thomas Broyer, who proposed http-cookie-auth (and a few others) on the http-auth group of IETF - in the github repo there are some more tests wrt browser support and rational.

It basically seems that we have a very reasonable and useful way forward that simplifies our codebase and simplifies the responses that we give - by interpreting the 401 error code in a way that is actually more logical.

(browsers show the html payload when they get a WWW-Authenticate type that they don't understand - which is pretty logical really)

-- SvenDowideit - 12 Feb 2012

My current commitment is only to the patch described at the head of this topic, but I'm willing to extend that to 401 for all requests if it's deemed appropriate. I understand the arguments for 401 for all requests, but I'm unclear what the arguments against it are. As I recall the discussions, the arguments raised against it were:

  1. 401 is reserved for browser auth (Basic and Digest)
    • I believe I disproved this through reference to the RFC's
    • the RFC http-auth people also seem to think that interpretation is wrong. SD
    • additionally, there's http-cookie-auth which dissagrees strongly - the objection to it has more to do with not liking to create an RFC that is insecure (as cookie auth is dodgey at best). SD
  2. No-one else does it that way
    • Again, disproved; OpenAuth and cookie-auth raised as counter-examples
    • http-cookie-auth draft, NTLM, and lots of other non-trivial history shows that 401 is the right thing to do. SD
  3. The solution makes XHR a special case.
    • My proposal certainly does that; extension to all requests does not.
    • why would it be desirable to have a special case? SD
  4. The proposal conflicts with the proposed standard http://tools.ietf.org/html/draft-oiwa-http-auth-extension-00
    • RFCs are notoriously hard to read, but by my reading the proposal conflicts less than the current implementation.
There is one other concern I might have; that we are relying on the goodwill of browsers w.r.t the handling of a 401. Experiment indicates that all current browsers ignore a 401 with an unrecognised auth method in the WWW-Authenticate header, simply displaying any HTML content of the page. However there are no standards for browsers w.r.t the handling of a 401 with unrecognised auth method. There is a working IETF proposal to implement a new auth method "HTML" that will support forms-based authentication (see http://blog.whatwg.org/this-week-in-html-5-episode-14). If we accept that this is likely to become a standard at some point (it seems likely) then the browsers are unlikely to change the way they handle 401 until it does.
  • I kinda get the feeling that draft hasn't a hope in hell SD
Which leaves us in something of a quandry. Do we implement forms authentication as described in the standards proposal? If we do, and the final standard conflicts with the current proposal, we are left with a problem. On the flip side, if we implement WWW-Authenticate: Somethingelse as I proposed, when the standard kicks in we will be out of step.

My personal view is that the standards world is in such confusion, and moves so slowly, that we can re-adapt to any emergent standard quickly enough, and should move ahead with 401+WWW-Authenticate: Foswiki for all auth failures.

A final note; if we are able to agree with a 401 response to all auth failures, then the patch I gave is not needed, as the standard response is a 401. We already have the ?username=;password= mechanism for passing back auth from JS. I would concur with Paul that this should be re-implemented using headers, but that is not part of this proposal.

-- CrawfordCurrie - 12 Feb 2012

  • I'd be quite happy if we do it for all requests, not just XHR.
  • George pointed out DigestJ, which is another application of non-standard WWW-Authenticate: http://code.google.com/p/digestj/wiki/DigestJ
  • Will we use Foswiki rather than one of the WWW-Authenticate schemes nominated in the draft RFCs, principally because we haven't made any attempt to be compatible with them? I think that's okay, but I am worried that Foswiki won't be enough of a hint for clients to figure out whether George's nifty new bcrypt thing is in use
-- PaulHarvey - 12 Feb 2012

I've removed my concern, as long as the commitment under this proposal is to handle the XHR 401 issues.

I suggest that this proposal be forked into something like ModernizeTemplateAuth to continue with:
  • Moving all auth requests to 401 responses
    • It probably ought to be configurable in the event that there are browsers or applications that cannot handle the 401.
  • Adding a JavaScript component possibly derived from DigestJ
It appears that 1.1.5 is moving towards being a security enhancement focused release. I'd support including a redesigned TemplateAuth effort as another major security improvement in 1.1.5.

-- GeorgeClark - 12 Feb 2012

Somewhere I saw someone suggest that we'd be relying on the goodwill of browsers to show the html payload for an WWW-Authenticate - I suspect thats not the case - if a browser doesn't know what to do with a 401 auth type, it has the choice to either display the html payload, or do nothing - which would be pretty woeful smile

I think Crawford summarized the possible objections to using 401+http-cookie-auth everywhere and Paul and I replied with what why we feel those are more flawed than consistency.

Talking to George on IRC, I think there's an unspoken assumption here:

this change to Template Auth makes it consistent with Apache Auth

for example - if a user goes to /bin/login the status code shall be the same as going to /bin/logon - 401. the difference would be the WWW-Authenticate value would be something the browser does not implement, so its fallback would be to show the html payload.

additionally, to avoid our spending time designing anything on the Protocol level, we follow the http-cookie-auth draft smile

-- SvenDowideit - 12 Feb 2012

A very simple change to Foswiki::LoginManager::TemplateLogin implements the 401 and it appears to work without the redirect to bin/login.
diff --git a/core/lib/Foswiki/LoginManager/TemplateLogin.pm b/core/lib/Foswiki/LoginManager/TemplateLogin.pm
index 928313a..5fd4db4 100755
--- a/core/lib/Foswiki/LoginManager/TemplateLogin.pm
+++ b/core/lib/Foswiki/LoginManager/TemplateLogin.pm
@@ -81,21 +81,17 @@ sub forceAuthentication {
     unless ( $session->inContext('authenticated') ) {
         my $query = $session->{request};
+        my $response = $session->{response};
-        # Redirect with passthrough so we don't lose the original query params
-        my $url = $session->getScriptUrl( 0, 'login' );
-        # We use the query here to ensure the original path_info
-        # from the request gets through to the login form. See also
-        # PATH_INFO below.
-        $url .= Foswiki::urlEncode( $query->path_info() );
+        $response->header( -status => 401,
+           -WWW_Authenticate => "HTML" );
             -name  => 'foswiki_origin',
             -value => _packRequest($session)
-        $session->redirect( $url, 1 );    # with passthrough
+        $this->login( $query, $session );
         return 1;
     return 0;

I suspect that this could be tweaked to allow the userid and password parameters to be passed to the script and accepted by the call to login. Rather than returning the login template, it could complete the login.

The WWW_Authenticate header in the above snip is incorrect - it needs to be structured with the challenge HTML and correct form field definitions.

One more suggestion - if we change TemplateLogin.pm - we should consider shipping a TemplateLogin/Redirecting.pm as a deprecated handler to allow sites to choose the old method.

-- GeorgeClark - 13 Feb 2012

George, that was exactly the change I was intending to make when I offered to support 401 in all responses :-). The challenge "HTML" overlaps with the forms-based authentication standards proposal I linked above, so is not IMHO a good choice. In the interests of not conflicting with emergent standards, I suggest we steer clear and follow Paul's Foswiki suggestion. I think it needs to be Foswiki realm=id to separate realms in the case that a single url root serves different Foswiki instances. RFC2617 dictates that support for the realm parameter is compulsory. _"The realm directive (case-insensitive) is required for all authentication schemes that issue a challenge. The realm value (case-sensitive), in combination with the canonical root URL (the absoluteURI for the server whose abs_path is empty; see section 5.1.2 of [2]) of the server being accessed, defines the protection space."_

I'd really like to finish this off by providing a login route using headers rather than the ?username=;password=;= route. According to RFC2617 this has to be done via the Authorization header. Before anyone objects that this is "reserved", please note that for JS to authenticate in an ApacheAuth environment it has to send this header. My only concern here is that the Authorization header may not get past the processing in the Web server, so may need to be duplicated in another header.

-- CrawfordCurrie - 13 Feb 2012

Fixed the commitment date, which was wrong.

Accepted proposal.

-- CrawfordCurrie - 16 Feb 2012
Topic revision: r15 - 05 Jul 2015, GeorgeClark - This page was cached on 26 May 2016 - 14:43.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License