As I mentioned in the immediately prior post in this series, the way to avoid having the browser prompt for credentials while using a Selenium test is by supplying the correct information in the Authorization header. Since Selenium's focus is automating the browser as close to how a user does so as possible, there's not a built-in way to examine or modify the headers. However, Selenium does make it very easy to configure the browser being automated to use a web proxy. A web proxy is a piece of software that stands between your browser and any request made of a web server, and can be made to examine, modify, or even block requests based on any number of rules. When configured to use a proxy, every request made by your browser flows through the proxy. Many businesses use proxies to ensure that only authorized resources are being accessed via business computers, or making sure that requests only come from authorized computers, or any number of other legitimate business purposes.
How do you configure your browser to use a proxy with Selenium? The code looks something like this:
Since we're Selenium users, we'll be using a proxy that allows us to programmatically start and stop it, and hook into the request/response chain via our code, and modify the results in order to interpret and replace the headers as needed. Any number of proxies could be used in this project. Many Selenium users have had great success using BrowserMob Proxy, or there are commercial options like Fiddler. Since I personally prefer FOSS options, and don't want to leave the .NET ecosystem, for our examples here, we'll be using BenderProxy. Here's the code for setting that up.
Now, how do we wire up the proper processing to mimic the browser's processing of an authentication prompt? We need to implement the addition of an Authorization header that provides the correct value, for the authentication scheme requested by the server. BenderProxy's OnResponseReceived handler happens after the response has been received from the web server, but before it's forwarded along to the browser for rendering. That gives us the opportunity to examine it, and resend another request with the proper credentials in the proper format. We're using the Basic authentication scheme in this example, and once again using The Internet sample application. Here's the code for the method:
Running the code, we'll see that when the Selenium code is run, the browser will show the authorized page, as we intended. As you can tell from the implementation code, Basic authentication is pretty simple, sending the Base64 encoding of "userName:passsword". Its simplicity is also one reason it's not used very often, as it sends the credentials across the wire, essentially in clear text. There are other, more secure authentication schemes available, and they can be automated in similar ways. The trick is knowing how to specify the value for the Authentication header. In the next post in the series, we'll look at another authentication mechanism, and how to handle something a little more complicated.
How do you configure your browser to use a proxy with Selenium? The code looks something like this:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Assumes the proxy is running on localhost, on port 9999. | |
Proxy proxy = new Proxy(); | |
proxy.HttpProxy = "127.0.0.1:9999"; | |
// Example uses Chrome, but can be adapted to any other browser. | |
ChromeOptions options = new ChromeOptions(); | |
options.Proxy = proxy; | |
// It would be strongly recommended to use a factory pattern | |
// to generate IWebDriver instances, but for purposes of this | |
// example, we'll just hard-code it for now. | |
IWebDriver driver = new ChromeDriver(option); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Set up BenderProxy, and grab the port | |
HttpProxyServer proxyServer = new HttpProxyServer("localhost", new HttpProxy()); | |
proxyServer.Start().WaitOne(); | |
int proxyPort = proxyServer.ProxyEndPoint.Port; | |
// Set up the WebDriver Proxy object, and the browser | |
// options to use it. | |
Proxy proxy = new Proxy(); | |
proxy.HttpProxy = string.Format("127.0.0.1:{0}", proxyPort); | |
FirefoxOptions options = new FirefoxOptions(); | |
options.Proxy = proxy; | |
// This is how to hook up the processing of HTTP requests/ | |
// responses with BenderProxy. ProcessBasicResponse will | |
// be an Action<ProcessingContext> (a method, returning void, | |
// taking a BenderProxy ProcessingContext as an argument). | |
proxyServer.Proxy.OnResponseReceived = ProcessBasicResponse; | |
IWebDriver driver = new FirefoxDriver(@"C:\Projects\WebDriverServers", options); | |
try | |
{ | |
driver.Url = "http://the-internet.herokuapp.com/basic_auth"; | |
} | |
finally | |
{ | |
driver.Quit(); | |
proxyServer.Stop(); | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
private static void ProcessBasicResponse(ProcessingContext context) | |
{ | |
string userName = "admin"; | |
string password = "admin"; | |
string authTypeMarker = "Basic "; | |
// Only do any processing on the response if the response is 401, | |
// or "Unauthorized". | |
if (context.ResponseHeader != null && context.ResponseHeader.StatusCode == 401) | |
{ | |
// Read the headers from the response and finish reading the response | |
// body, if any (implementation of ReadFromStream() is left as an | |
// exercise for the reader). | |
context.ServerStream.ReadTimeout = 5000; | |
context.ServerStream.WriteTimeout = 5000; | |
StreamReader reader = new StreamReader(context.ServerStream); | |
if (context.ResponseHeader.EntityHeaders.ContentLength != 0) | |
{ | |
string drainBody = ReadFromStream(reader); | |
} | |
// We do not want the proxy to do any further processing after | |
// handling this message. | |
context.StopProcessing(); | |
// To be fully correct, should check the header for the auth type. | |
string authHeader = context.ResponseHeader.WWWAuthenticate; | |
// This is the generation of the HTTP Basic authorization header value. | |
string basicAuthHeaderValue = string.Format("{0}:{1}", userName, password); | |
string encodedHeaderValue = Convert.ToBase64String(Encoding.ASCII.GetBytes(basicAuthHeaderValue)); | |
context.RequestHeader.Authorization = authTypeMarker + encodedHeaderValue; | |
// Resend the request (with the Authorization header) to the server | |
// using BenderProxy's HttpMessageWriter. | |
HttpMessageWriter writer = new HttpMessageWriter(context.ServerStream); | |
writer.Write(context.RequestHeader); | |
// Get the authorized response, and forward it on to the browser, using | |
// BenderProxy's HttpHeaderReader and support classes. | |
HttpHeaderReader headerReader = new HttpHeaderReader(reader); | |
HttpResponseHeader header = new HttpResponseHeader(headerReader.ReadHttpMessageHeader()); | |
string body = ReadFromStream(reader); | |
Stream bodyStream = new MemoryStream(Encoding.UTF8.GetBytes(body)); | |
new HttpResponseWriter(context.ClientStream).Write(header, bodyStream, bodyStream.Length); | |
} | |
} |
Thanks a lot for a great post, and for .NET BrowserMob proxy alternative!
ReplyDeleteThere is one question I'm wondering about. Is there any way to make it work with Selenium Grid? Or with services like BrowserStack?
Do I understand correctly, that in this case machine, executing test code, should have public IP address - so that node would be able to connect to proxy server that we created?
Yes, the machine on which the browser is executing must be able to see the machine on which the proxy is running. Remember, use of a proxy is a browser feature. You can use one outside the context of Selenium WebDriver. Indeed, from the context of .NET test code, it's probably best that the proxy server be running on the same machine as the test code itself, particularly if you want to take advantage of being able to analyze or edit requests and responses. There is nothing, however, preventing you from packaging up your proxy execution code, and running on some other machine. The only issue would be how to communicate with that proxy from your test code.
Delete