Implementing HTTP Status Codes in WebDriver, Part 1: Challenge Accepted

A while back, I wrote a post that discussed at length why HTTP status codes are not present in the WebDriver API. Furthermore, the post went on to explain why I believe they're not needed in the API, and that there are other tools better suited to retrieving this particular piece of esoterica. Since I wrote that article, other Selenium contributors have written about the same topic. The general premise of those blog posts and mine is that using a proxy is the proper way to capture the status code if you actually require it.

Nevertheless, the issue in the Selenium issue tracker that was the inspiration for those blog posts continues to receive comments, most of them vehemently opposed to the decision of the Selenium development team. The decision has been called "complete nonsense," "silly," "condescending," and "simply defective," among other choice phrases. My colleagues on the Selenium project have posted code samples that show how to effectively use a proxy with Selenium to solve this problem, and the response to those samples has been that they aren't detailed enough.

Alright, fine. Time to put my proverbial money where my mouth is. I recently looked into what it would take to actually implement a proxy solution, with correct return of HTTP status codes, including writing all of the code necessary to extract it. It doesn't take that much, as it turns out. Once I'd settled on a technology to use, I had a full working example in about 4 hours. Let's take a look at how this would work.

In my example, I decided to use the Mozilla website, http://www.mozilla.org/, as my test. I settled on this because the site isn't likely to disappear anytime soon, and as currently written, it nicely demonstrates some of the issues inherent in getting HTTP status codes. Please note that I don't own the website, so it's possible that these examples could break at any time after this writing; at some point, I'll look at creating a standalone website that illustrates the same concepts. I'm also going to be using the WebDriver .NET bindings, and specifically, version 2.35.0 of the .NET bindings. For the proxy component, I decided to use Eric Lawrence's (now Telerik's) excellent Fiddler proxy.

Let's talk for a moment about why I chose Fiddler. First, I'm a .NET guy, and I try to look for solutions that don't require another runtime (like Java, Ruby, or Python) if possible. Second, Fiddler offers me the FiddlerCore component, which allows me to use an API-only version of Fiddler, and programmatic access to all of the proxy's settings and data. The API may be a little less polished than other .NET component APIs, but it does use an event-driven model, which appeals to me as a .NET developer. While Fiddler isn't open-source, it is free to use, with no feature restrictions based on free vs. paid use. With all of that in mind, let's begin. Here's the basic code that I want to run, using the standard WebDriver API:

private static void TestStatusCodes(IWebDriver driver)
{
    // Using Mozilla's main page, because it demonstrates some of
    // the potential problems with HTTP status code retrieval, and
    // why there is not a one-size-fits-all approach to it.
    string url = "http://www.mozilla.org/";
    driver.Navigate().GoToUrl(url);

    string elementId = "firefox-promo-link";
    IWebElement element = driver.FindElement(By.Id(elementId));
    element.Click();

    // Demonstrates navigating to a 404 page.
    url = "http://www.mozilla.org/en-US/doesnotexist.html";
    driver.Navigate().GoToUrl(url);
}

I'll be running this method from within a console application, with the main method looking something like this:

static void Main(string[] args)
{
    // Eventually, we will use different browsers to prove this
    // solution works cross-browser, but for now, we will use
    // Firefox only.
    IWebDriver driver = new FirefoxDriver();

    TestStatusCodes(driver);

    driver.Quit();

    Console.WriteLine("Complete! Press <Enter> to exit.");
    Console.ReadLine();
}

Let's look at how to integrate Fiddler in this solution. Starting the proxy server couldn't be easier. We'll create a method to do this. One thing to note in the method is that we can either specify a port for the proxy to listen on, or let Fiddler pick one for us.

private static int StartFiddlerProxy(int desiredPort)
{
    // We explicitly do *NOT* want to register this running Fiddler
    // instance as the system proxy. This lets us keep isolation.
    Console.WriteLine("Starting Fiddler proxy");
    FiddlerCoreStartupFlags flags = FiddlerCoreStartupFlags.Default &
                                    ~FiddlerCoreStartupFlags.RegisterAsSystemProxy;

    FiddlerApplication.Startup(desiredPort, flags);
    int proxyPort = FiddlerApplication.oProxy.ListenPort;
    Console.WriteLine("Fiddler proxy listening on port {0}", proxyPort);
    return proxyPort;
}

Technically speaking, we probably don't need to shut down the proxy, since it's the last thing we do before our main method exits, but we're going to be a good citizen and shut it down anyway.

private static int StopFiddlerProxy()
{
    Console.WriteLine("Shutting down Fiddler proxy");
    FiddlerApplication.Shutdown();
}

All that remains is to hook up an event handler so that we can analyze the traffic that comes through the proxy, and to make the Firefox driver aware of the proxy. We can do those things within the context of our main method. After all of these, the final main method looks like this:

static void Main(string[] args)
{
    // Note that we're using a desired port of 0, which tells
    // Fiddler to select a random available port to listen on.
    int proxyPort = StartFiddlerProxy(0);

    // Hook up the event for monitoring proxied traffic.
    FiddlerApplication.AfterSessionComplete += delegate(Session targetSession)
    {
        Console.WriteLine("Requested resource from URL {0}",
                          targetSession.fullUrl);
    };

    // We are only proxying HTTP traffic, but could just as easily
    // proxy HTTPS or FTP traffic.
    OpenQA.Selenium.Proxy proxy = new OpenQA.Selenium.Proxy();
    proxy.HttpProxy = string.Format("127.0.0.1:{0}", proxyPort);

    // Eventually, we will use different browsers to prove this
    // solution works cross-browser, but for now, we will use
    // Firefox only.
    FirefoxProfile profile = new FirefoxProfile();
    profile.SetProxyPreferences(proxy);
    IWebDriver driver = new FirefoxDriver(profile);

    TestStatusCodes(driver);

    driver.Quit();

    Console.WriteLine("Complete! Press <Enter> to exit.");
    Console.ReadLine();
}

When we run our console application, we see something like this:

Starting Fiddler proxy
Fiddler proxy listening on port 62492
Navigating to http://www.mozilla.org/
Requested resource from URL http://www.mozilla.org/
Requested resource from URL http://mozorg.cdn.mozilla.net/media/css/tabzilla-min.css?build=c2a3f7a
Requested resource from URL http://mozorg.cdn.mozilla.net/media/js/site-min.js?build=c2a3f7a
Requested resource from URL http://mozorg.cdn.mozilla.net/media/css/responsive-min.css?build=c2a3f7a
Requested resource from URL http://mozorg.cdn.mozilla.net/media/img/favicon.ico
Requested resource from URL http://www.mozilla.org/en-US/

[... 
Many resources deleted for brevity
...]

Clicking on element with ID firefox-promo-link
Requested resource from URL http://mozorg.cdn.mozilla.net/media/fonts/Vollkorn-Regular-webfont.woff
Requested resource from URL http://mozorg.cdn.mozilla.net/media/fonts/Vollkorn-Bold-webfont.woff
Requested resource from URL http://mozorg.cdn.mozilla.net/media/img/home/promo/flicks/760.jpg
Requested resource from URL http://mozorg.cdn.mozilla.net/media/img/home/promo/android/760.jpg?2013-06
Requested resource from URL http://mozorg.cdn.mozilla.net/media/img/home/promo/makerparty/760.jpg
Navigating to http://www.mozilla.org/en-US/doesnotexist.html
Requested resource from URL http://www.mozilla.org/firefox/
Requested resource from URL http://www.mozilla.org/en-US/firefox/
Requested resource from URL http://mozorg.cdn.mozilla.net/media/css/firefox_fx-min.css?build=c2a3f7a
Requested resource from URL http://mozorg.cdn.mozilla.net/media/img/firefox/template/header-logo.png?2013-06
Requested resource from URL http://www.mozilla.org/en-US/firefox/fx/ 

[... 
Many resources deleted for brevity
...]

Shutting down Fiddler proxy
Complete! Press <Enter> to exit. 

Obviously, this particular example doesn't get us to our desired goal just yet. However, it does allow us to hook up a proxy. Next time, I'll show you how we can refine this solution to actually extract those status codes.