Handling Authentication Requests with Selenium - Part 3: Beyond Basic Authentication

In the last post in this series, we saw the general procedure for handling authentication requests with Selenium and a web proxy:

• Start the programmable proxy
• Start a Selenium session configuring the browser to use the proxy
• Wire up a method to intercept the 401 Unauthorized response
• Use the method to resend the request with the correct Authorization header value

As we noted previously, the use of the Basic HTTP authentication scheme is rather weak. There are other authentication schemes that don't require the sending of a password in plain text over the wire. One such case is HTTP Digest authentication. Let's see what that looks like. First, let's navigate to a page that implements Digest authentication, and examine what we see. As before, we'll use the hosted version of The Internet at http://the-internet.herokuapp.com/

Browser sends:
GET http://the-internet.herokuapp.com/digest_auth HTTP/1.1
Host: the-internet.herokuapp.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Browser receives back:
HTTP/1.1 401 Unauthorized
Connection: keep-alive
Content-Type: text/plain
Content-Length: 0
Www-Authenticate: Digest realm="Protected Area", nonce="MTU1ODkwNDI2MyBkYjYzMTA0ZTY0NmZjNmZhNDljNzQ2ZGY0ZTc3NDM4OA==", opaque="610a2ee688cda9e724885e23cd2cfdee", qop="auth"
Server: WEBrick/1.3.1 (Ruby/2.2.5/2016-04-26)
Date: Sun, 26 May 2019 20:57:43 GMT
Via: 1.1 vegurTTP/1.1 200 OK 

Note the value of the WWW-Authenticate header, which is considerably more complex than in the Basic authentication scheme case. The algorithm for figuring out the correct value for the Authorization header is likewise much more complex, which, in the simplest case, involves getting the MD5 hash of the string "userName:realm:password", then the MD5 hash of the HTTP verb and the URL of the resource being requested, then getting the Base64-encoded string of those two hashes along with the "nonce" value send in the authenticate header.

Whew. That's an awful lot to keep straight. Probably a little too complicated to post the code for resolving all of the nuances of it within this blog post. So it's time to introduce a new library to add to our toolbox for calculating the authorization header value for any of a variety of authentication methods. That library is called PassedBall, and it's available both on GitHub and as a NuGet package. Since PassedBall supports Digest authentication, and using the same process as in our previous post, here's the implementation of the method to intercept and resend the HTTP request:

Now that we have a library and generic framework for the generation of arbitrary authentication schemes, we'll look at one last approach for authentication, one that uses connection semantics for authentication, NTLM authentication.