Handling Authentication Requests with Selenium - Part 3: Beyond Basic Authentication

In the last post in this series, we saw the general procedure for handling authentication requests with Selenium and a web proxy:

• Start the programmable proxy
• Start a Selenium session configuring the browser to use the proxy
• Wire up a method to intercept the 401 Unauthorized response
• Use the method to resend the request with the correct Authorization header value

As we noted previously, the use of the Basic HTTP authentication scheme is rather weak. There are other authentication schemes that don't require the sending of a password in plain text over the wire. One such case is HTTP Digest authentication. Let's see what that looks like. First, let's navigate to a page that implements Digest authentication, and examine what we see. As before, we'll use the hosted version of The Internet at http://the-internet.herokuapp.com/

Browser sends:
GET http://the-internet.herokuapp.com/digest_auth HTTP/1.1
Host: the-internet.herokuapp.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Browser receives back:
HTTP/1.1 401 Unauthorized
Connection: keep-alive
Content-Type: text/plain
Content-Length: 0
Www-Authenticate: Digest realm="Protected Area", nonce="MTU1ODkwNDI2MyBkYjYzMTA0ZTY0NmZjNmZhNDljNzQ2ZGY0ZTc3NDM4OA==", opaque="610a2ee688cda9e724885e23cd2cfdee", qop="auth"
Server: WEBrick/1.3.1 (Ruby/2.2.5/2016-04-26)
Date: Sun, 26 May 2019 20:57:43 GMT
Via: 1.1 vegurTTP/1.1 200 OK 

Note the value of the WWW-Authenticate header, which is considerably more complex than in the Basic authentication scheme case. The algorithm for figuring out the correct value for the Authorization header is likewise much more complex, which, in the simplest case, involves getting the MD5 hash of the string "userName:realm:password", then the MD5 hash of the HTTP verb and the URL of the resource being requested, then getting the Base64-encoded string of those two hashes along with the "nonce" value send in the authenticate header.

Whew. That's an awful lot to keep straight. Probably a little too complicated to post the code for resolving all of the nuances of it within this blog post. So it's time to introduce a new library to add to our toolbox for calculating the authorization header value for any of a variety of authentication methods. That library is called PassedBall, and it's available both on GitHub and as a NuGet package. Since PassedBall supports Digest authentication, and using the same process as in our previous post, here's the implementation of the method to intercept and resend the HTTP request:

	private static void ProcessDigestResponse(ProcessingContext context)
	{
	string userName = "admin";
	string password = "admin";
	string httpMethod = context.RequestHeader.Method;
	string hostHeaderValue = context.RequestHeader.Host;
	string requestUri = context.RequestHeader.RequestURI;
	int hostIndex = requestUri.IndexOf(hostHeaderValue);
	if (hostIndex >= 0)
	{
	requestUri = requestUri.Substring(hostIndex + hostHeaderValue.Length);
	}
	if (context.ResponseHeader != null && context.ResponseHeader.StatusCode == 401)
	{
	// Process only if the server is requesting Digest authentication.
	if (context.ResponseHeader.WWWAuthenticate.Contains(DigestGenerator.AuthorizationHeaderMarker))
	{
	// Read the headers from the response and finish reading the response
	// body, if any (implementation of ReadFromStream() is left as an
	// exercise for the reader).
	context.ServerStream.ReadTimeout = 5000;
	context.ServerStream.WriteTimeout = 5000;
	StreamReader reader = new StreamReader(context.ServerStream);
	if (context.ResponseHeader.EntityHeaders.ContentLength != 0)
	{
	string drainBody = ReadFromStream(reader);
	}
	// We do not want the proxy to do any further processing after
	// handling this message.
	context.StopProcessing();
	// Use the PassedBall library to read the WWW-Authenticate
	// header and generate the Authorization header value.
	string authHeader = context.ResponseHeader.WWWAuthenticate;
	DigestGenerator digest = new DigestGenerator(userName, password, httpMethod, requestUri, authHeader);
	context.RequestHeader.Authorization = digest.GenerateAuthorizationHeader();
	// Resend the request (with the Authorization header) to the server
	// using BenderProxy's HttpMessageWriter.
	HttpMessageWriter writer = new HttpMessageWriter(context.ServerStream);
	writer.Write(context.RequestHeader);
	// Get the authorized response, and forward it on to the browser, using
	// BenderProxy's HttpHeaderReader and support classes.
	HttpHeaderReader headerReader = new HttpHeaderReader(reader);
	HttpResponseHeader header = new HttpResponseHeader(headerReader.ReadHttpMessageHeader());
	string body = ReadFromStream(reader);
	Stream bodyStream = new MemoryStream(Encoding.UTF8.GetBytes(body));
	new HttpResponseWriter(context.ClientStream).Write(header, bodyStream, bodyStream.Length);
	}
	}
	}

view raw DigestAuthHeaderGeneration.cs hosted with ❤ by GitHub

Now that we have a library and generic framework for the generation of arbitrary authentication schemes, we'll look at one last approach for authentication, one that uses connection semantics for authentication, NTLM authentication.

Handling Authentication Requests with Selenium - Part 2: Using a Web Proxy for Basic Authentication

As I mentioned in the immediately prior post in this series, the way to avoid having the browser prompt for credentials while using a Selenium test is by supplying the correct information in the Authorization header. Since Selenium's focus is automating the browser as close to how a user does so as possible, there's not a built-in way to examine or modify the headers. However, Selenium does make it very easy to configure the browser being automated to use a web proxy. A web proxy is a piece of software that stands between your browser and any request made of a web server, and can be made to examine, modify, or even block requests based on any number of rules. When configured to use a proxy, every request made by your browser flows through the proxy. Many businesses use proxies to ensure that only authorized resources are being accessed via business computers, or making sure that requests only come from authorized computers, or any number of other legitimate business purposes.

How do you configure your browser to use a proxy with Selenium? The code looks something like this:

	// Assumes the proxy is running on localhost, on port 9999.
	Proxy proxy = new Proxy();
	proxy.HttpProxy = "127.0.0.1:9999";
	// Example uses Chrome, but can be adapted to any other browser.
	ChromeOptions options = new ChromeOptions();
	options.Proxy = proxy;
	// It would be strongly recommended to use a factory pattern
	// to generate IWebDriver instances, but for purposes of this
	// example, we'll just hard-code it for now.
	IWebDriver driver = new ChromeDriver(option);

view raw WebDriverWithProxy.cs hosted with ❤ by GitHub

Since we're Selenium users, we'll be using a proxy that allows us to programmatically start and stop it, and hook into the request/response chain via our code, and modify the results in order to interpret and replace the headers as needed. Any number of proxies could be used in this project. Many Selenium users have had great success using BrowserMob Proxy, or there are commercial options like Fiddler. Since I personally prefer FOSS options, and don't want to leave the .NET ecosystem, for our examples here, we'll be using BenderProxy. Here's the code for setting that up.

	// Set up BenderProxy, and grab the port
	HttpProxyServer proxyServer = new HttpProxyServer("localhost", new HttpProxy());
	proxyServer.Start().WaitOne();
	int proxyPort = proxyServer.ProxyEndPoint.Port;
	// Set up the WebDriver Proxy object, and the browser
	// options to use it.
	Proxy proxy = new Proxy();
	proxy.HttpProxy = string.Format("127.0.0.1:{0}", proxyPort);
	FirefoxOptions options = new FirefoxOptions();
	options.Proxy = proxy;
	// This is how to hook up the processing of HTTP requests/
	// responses with BenderProxy. ProcessBasicResponse will
	// be an Action<ProcessingContext> (a method, returning void,
	// taking a BenderProxy ProcessingContext as an argument).
	proxyServer.Proxy.OnResponseReceived = ProcessBasicResponse;
	IWebDriver driver = new FirefoxDriver(@"C:\Projects\WebDriverServers", options);
	try
	{
	driver.Url = "http://the-internet.herokuapp.com/basic_auth";
	}
	finally
	{
	driver.Quit();
	proxyServer.Stop();
	}

view raw WebDriverWithProxy.cs hosted with ❤ by GitHub

Now, how do we wire up the proper processing to mimic the browser's processing of an authentication prompt? We need to implement the addition of an Authorization header that provides the correct value, for the authentication scheme requested by the server. BenderProxy's OnResponseReceived handler happens after the response has been received from the web server, but before it's forwarded along to the browser for rendering. That gives us the opportunity to examine it, and resend another request with the proper credentials in the proper format. We're using the Basic authentication scheme in this example, and once again using The Internet sample application. Here's the code for the method:

	private static void ProcessBasicResponse(ProcessingContext context)
	{
	string userName = "admin";
	string password = "admin";
	string authTypeMarker = "Basic ";
	// Only do any processing on the response if the response is 401,
	// or "Unauthorized".
	if (context.ResponseHeader != null && context.ResponseHeader.StatusCode == 401)
	{
	// Read the headers from the response and finish reading the response
	// body, if any (implementation of ReadFromStream() is left as an
	// exercise for the reader).
	context.ServerStream.ReadTimeout = 5000;
	context.ServerStream.WriteTimeout = 5000;
	StreamReader reader = new StreamReader(context.ServerStream);
	if (context.ResponseHeader.EntityHeaders.ContentLength != 0)
	{
	string drainBody = ReadFromStream(reader);
	}
	// We do not want the proxy to do any further processing after
	// handling this message.
	context.StopProcessing();
	// To be fully correct, should check the header for the auth type.
	string authHeader = context.ResponseHeader.WWWAuthenticate;
	// This is the generation of the HTTP Basic authorization header value.
	string basicAuthHeaderValue = string.Format("{0}:{1}", userName, password);
	string encodedHeaderValue = Convert.ToBase64String(Encoding.ASCII.GetBytes(basicAuthHeaderValue));
	context.RequestHeader.Authorization = authTypeMarker + encodedHeaderValue;
	// Resend the request (with the Authorization header) to the server
	// using BenderProxy's HttpMessageWriter.
	HttpMessageWriter writer = new HttpMessageWriter(context.ServerStream);
	writer.Write(context.RequestHeader);
	// Get the authorized response, and forward it on to the browser, using
	// BenderProxy's HttpHeaderReader and support classes.
	HttpHeaderReader headerReader = new HttpHeaderReader(reader);
	HttpResponseHeader header = new HttpResponseHeader(headerReader.ReadHttpMessageHeader());
	string body = ReadFromStream(reader);
	Stream bodyStream = new MemoryStream(Encoding.UTF8.GetBytes(body));
	new HttpResponseWriter(context.ClientStream).Write(header, bodyStream, bodyStream.Length);
	}
	}

view raw BasicResponseHeader.cs hosted with ❤ by GitHub

Running the code, we'll see that when the Selenium code is run, the browser will show the authorized page, as we intended. As you can tell from the implementation code, Basic authentication is pretty simple, sending the Base64 encoding of "userName:passsword". Its simplicity is also one reason it's not used very often, as it sends the credentials across the wire, essentially in clear text. There are other, more secure authentication schemes available, and they can be automated in similar ways. The trick is knowing how to specify the value for the Authentication header. In the next post in the series, we'll look at another authentication mechanism, and how to handle something a little more complicated.

Handling Authentication Requests with Selenium - Part 1: How Does Browser Authentication Work Anyway?

In order to understand how to use browser technologies to automate pages that use some form of authentication, it is useful to know what happens when you browse to such a page. What's actually happening when your browser prompts you for some form of credentials, usually a user name and password, before it will let you access a given resource on the web?

At the risk of dropping down to a ridiculously low level, let's talk about how browsers transfer data for browsing websites. First, an obligatory disclaimer. I'm going to deliberately gloss over using pages served via secure HTTP ("https"), and I'm going to ignore mostly-binary protocols like HTTP/2 for this series. Those items, while important, and may impact the outcomes you see here, are beyond the scope of this series.

Most of the time, a browser is using the Hypertext Transfer Protocol (or HTTP) to communicate with a given web server. When you type in a URL in your browser's address bar, your browser sends off an HTTP request (that's what the "http://" means at the beginning of the URL), and receives a response from the server. For the following examples, we'll be using Dave Haeffner's excellent Selenium-focused testing site, The Internet, which is designed to provide examples of challenging things a user might encounter when automating web pages with Selenium, and a hosted version of which is available at http://the-internet.herokuapp.com. Here's what a typical exchange might look like:

Browser sends:
GET http://the-internet.herokuapp.com/checkboxes HTTP/1.1
Host:the-internet.herokuapp.com
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language:en-US,en;q=0.5
Accept-Encoding:gzip, deflate
Connection:keep-alive
Upgrade-Insecure-Requests:1

Browser receives back:
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: text/html;charset=utf-8
Content-Length: 2008
Server: WEBrick/1.3.1 (Ruby/2.2.5/2016-04-26)
Date: Thu, 23 May 2019 23:44:54 GMT
Via: 1.1 vegur

<body of HTML page here>

This is what happens for virtually every time a browser makes a request for a resource. The important thing to note is in that first line of the response. The "200 OK" bit means that the server had the resource and was sending it in response to the request. Now let's look at a request for a resource that is protected by authentication:

Browser sends:
GET http://the-internet.herokuapp.com/basic_auth HTTP/1.1
Host: the-internet.herokuapp.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Browser receives back:
HTTP/1.1 401 Unauthorized
Connection: keep-alive
Content-Type: text/html;charset=utf-8
Www-Authenticate: Basic realm="Restricted Area"
Content-Length: 15
Server: WEBrick/1.3.1 (Ruby/2.2.5/2016-04-26)
Date: Thu, 23 May 2019 23:52:24 GMT
Via: 1.1 vegur

Note the all-important first line of the response, which says "401 Unauthorized". That tells us that we have a page that requires authentication. Note that if you asked your browser to browse to the page http://the-internet.herokuapp.com/basic_auth, you would have been prompted for a user name and password. Note in the response the line that says Www-Authenticate: Basic realm="Restricted Area". That tells the browser that the "Basic" authentication scheme is expected, and that the user's user name and password are required, and so the browser prompts you, and then it re-sends the request to the server, but with an additional header. If you used the proper credentials for the aforementioned URL (user name: admin, password: admin), you'd see something like the following:

Browser sends:
GET http://the-internet.herokuapp.com/basic_auth HTTP/1.1
Host: the-internet.herokuapp.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Authorization: Basic YWRtaW46YWRtaW4=

Browser receives back:
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: text/html;charset=utf-8
Content-Length: 1643
Server: WEBrick/1.3.1 (Ruby/2.2.5/2016-04-26)
Date: Thu, 23 May 2019 23:59:31 GMT
Via: 1.1 vegur

<body of HTML page here>

Clearly, that additional header that says Authorization: Basic YWRtaW46YWRtaW4= tells us that the browser must've done something with those credentials we gave it. If only we had a way to intercept the unauthorized response, calculate what needs to go into that authorization header, and resend the request before the browser had the chance to prompt us for credentials, we'd be golden. As luck (and technology) would have it, we do have exactly that ability, by using a web proxy. Every browser supports proxies, and Selenium makes it incredibly easy to use them with browsers being automated by it. The next post in this series will outline how to get that set up and working with Selenium.

Handling Authentication Requests with Selenium - Prologue

When you've been using Selenium for UI-based testing by using browser automation as long as I have, you occasionally see questions repeat themselves. Nearly any website of any complexity has functionality that relies on user authentication. Nowadays, that's most often done with some sort of form-based UI, which creates a session that gets tracked by the browser via a cookie. The variations of those are wide, but they don't require any additional tooling to handle via Selenium.

However, every now and again, someone will post to one of the Selenium mailing lists or on the IRC/Slack channel asking about a site they have to automate that relies on the browser itself asking for the credentials with which to attempt authentication. Selenium is very good at automating pages within a browser, but makes little or no attempt at automating the parts of the browser outside the page being displayed, like file download selection dialogs, print dialogs, or, and most relevant here, browser-displayed credential dialogs. This usually means that the user tries to find another way to manipulate the dialog, and that often means a language-specific tool (like Java's Robot class) or a platform-specific one (like AutoIt), or an approach that once worked with browser, but has now been deprecated and disallowed by nearly all as a security risk (putting the user name and password directly in the URL). The challenge with these approaches is that they rarely scale well, and almost never work correctly in the remote or grid case.

This happens often enough that I'm going to start a series of posts about authentication, and how to effectively automate it with Selenium, without resorting to single-language or single-platform utilities. The last time I posted a series of blog posts about how to accomplish something using Selenium combined with other tools, one of the comments I got was that they were "not impressed it [took] three blog articles to explain how to accomplish" the task.

The blog post series in question could easily have been done in one post, with a simple code example, but without any explanation of what the code was actually doing. I don't believe that approach is worthwhile, and is actually detrimental to learning.

So let me lay this out ahead of time. The content of this post series could be done in a single post. It would be extraordinarily long, and a lot of it would end up saying, "go look at the sample code to get the full picture." I'd rather not take that approach. I'd much rather take my time, and at least attempt to give the relevant details in smaller, more quickly digested chunks. If you find that approach lacking, and would rather "just give me teh codez," I'll respectfully suggest heading elsewhere. As I add posts to the series, I'll try to keep an updated list of the parts of the series at the bottom of each post so that you can jump to the section you need.

One last thing, I'll be showing code using C#. Assuming you have similar libraries available in your language of choice, the code can be ported to other languages and libraries. Doing so in this space will be beyond the scope of this post series. Also, I'll be using a more verbose coding style than is possible with modern versions of C#. This is a stylistic choice for explicitness and clarity; you're welcome to use more modern syntax in your own code.

Posts in this series:
Prologue: This post
Part 1: How Does Browser Authentication Work Anyway?
Part 2: Using a Web Proxy for Basic Authentication
Part 3: Beyond Basic Authentication 
Part 4: NTLM Authentication  
Epilogue: Final Thoughts