Handling Authentication Requests with Selenium - Part 4: NTLM Authentication

Now that we've created a thorough intellectual framework for how to handle authentication requests using Selenium in combination with a web proxy, and thanks to our last post, we can handle more than Basic authentication, let's take things a step further, and see how you can use Selenium in automating pages secured with NTLM authentication. Before we can do that, though, we need to have an understanding of how NTLM authentication differs from the previous types of authentication we've used before.

NTLM authentication is a Microsoft-developed technology, originally implemented in the company's IIS web server product. It's not widely used on the public internet, but it does integrate nicely with things like Active Directory, so it can be quite useful for web applications used on company intranets that require security based on Active Directory credentials. This means that to provide sample code, we'll need to have a few things in place first. First, we'll need a test website that we can run locally, running on a server that implements NTLM authentication. Since we're working in C# in this series, we can create an ASP.NET Core web project to do that.

Second, we'll also need to host the application using Windows. Even though the ASP.NET Core project can run against .NET Core, and that can run on platforms other than Windows, we'll need to actually run on Windows to take advantage of NTLM authentication, unless we want to introduce a ton of complexity with Active Directory domains and the like (which we don't for this post).

Finally, most browsers bypass the use of a proxy when running strictly on localhost. This means that if you're running things all on the same system, you'll need to either configure the browser not to do this, or trick it into thinking the site the browser is connecting to isn't the local machine. The latter is far easier, since it only involves adding a line to the Windows hosts file (located at %WinDir%\System32\drivers\etc\hosts). On my test system, I've redirected www.seleniumhq-test.test to 127.0.0.1 by using the hosts file, and the sample code will reflect this.

NTLM authentication is a challenge-response based authentication scheme, and it differs from other HTTP authentication schemes in that it authenticates a connection, not an individual request. This means that the browser and server must support so-called "keep-alive," or persistent TCP connections between them. It also means that our proxy has to support persistent TCP connections, and must allow us to use that exact connection for making the requests. Fortunately, the proxy we've been using so far, BenderProxy, does support this.

The challenge-response mechanism used is complicated. Very complicated. So again, we'll be using the PassedBall library to parse authentication headers and generate authorization responses. It also requires multiple request/response round trips to perform the authentication handshake. Here's the implementation code for handling the NTLM authentication challenge for a sample site hosted on our local host machine:

	public static void ProcessNtlmResponse(ProcessingContext context)
	{
	string userName = "NtlmAuthTestUser";
	string password = "NtlmAuthTestP@ssw0rd!";
	if (context.ResponseHeader != null && context.ResponseHeader.StatusCode == 401)
	{
	// Only process requests for localhost or the redirected-
	// via-hosts-file-entry host, and where NTLM auth is requested.
	List<string> candidateUrls = new List<string>() { "localhost", "www.seleniumhq-test.test" };
	if (candidateUrls.Contains(context.RequestHeader.Host) && context.ResponseHeader.WWWAuthenticate != null && context.ResponseHeader.WWWAuthenticate.Contains(NtlmGenerator.AuthorizationHeaderMarker))
	{
	// Read the headers from the response and finish reading the response
	// body, if any (implementation of ReadFromStream() is left as an
	// exercise for the reader).
	context.ServerStream.ReadTimeout = 5000;
	context.ServerStream.WriteTimeout = 5000;
	StreamReader reader = new StreamReader(context.ServerStream);
	string drainBody = ReadFromStream(reader);
	// We do not want the proxy to do any further processing after
	// handling this message.
	context.StopProcessing();
	// Generate the initial message (the "type 1" or "Negotiation" message")
	// and get the response, using BenderProxy's HttpMessageWriter and
	// HttpHeaderReader and support classes.
	NtlmNegotiateMessageGenerator type1 = new NtlmNegotiateMessageGenerator();
	context.RequestHeader.Authorization = type1.GenerateAuthorizationHeader();
	HttpMessageWriter writer = new HttpMessageWriter(context.ServerStream);
	writer.Write(context.RequestHeader);
	HttpHeaderReader headerReader = new HttpHeaderReader(reader);
	HttpResponseHeader challengeHeader = new HttpResponseHeader(headerReader.ReadHttpMessageHeader());
	string authHeader = challengeHeader.WWWAuthenticate;
	string challengeBody = ReadFromStream(reader);
	if (!string.IsNullOrEmpty(authHeader))
	{
	// If a proper message was received (the "type 2" or "Challenge" message),
	// parse it, and generate the proper authentication header (the "type 3"
	// or "Authorization" message).
	NtlmChallengeMessageGenerator type2 = new NtlmChallengeMessageGenerator(authHeader);
	NtlmAuthenticateMessageGenerator type3 = new NtlmAuthenticateMessageGenerator(null, null, userName, password, type2);
	context.RequestHeader.Authorization = type3.GenerateAuthorizationHeader();
	writer.Write(context.RequestHeader);
	// Get the authorized response from the server, and forward it on to
	// the browser.
	HttpResponseHeader header = new HttpResponseHeader(headerReader.ReadHttpMessageHeader());
	string body = ReadFromStream(reader);
	Stream bodyStream = new MemoryStream(Encoding.UTF8.GetBytes(body));
	new HttpResponseWriter(context.ClientStream).Write(header, bodyStream, bodyStream.Length);
	}
	}
	}
	}

view raw NtlmAuthHeaderGeneration.cs hosted with ❤ by GitHub

Note carefully that the initial 401 Unauthorized response may contain multiple WWW-Authenticate headers, so one may need to make sure the proper one is being used to interpret the response. Browsers, when faced with this, will usually choose what they perceive to be the "strongest" authentication method. In our case, we need to do that determination for ourselves.

We'll wrap up this series with one more post, summing everything up.