Implementing HTTP Status Codes in WebDriver, Part 3: Fit and Finish

This is the final part in my blog series about implementing retrieval of HTTP status codes in WebDriver. In Part 1, I demonstrated the basic premise of enabling use of a proxy to monitor HTTP traffic between the browser and the server providing the pages. In Part 2, I expanded that solution to actually inspect the traffic for the HTTP status codes. In this part, we'll be finishing off the solution by demonstrating how it works cross-browser, and using a few more tweaks to make the solution a little more elegant.

First, let's tackle the cross-browser cases. We'll start by creating a factory and an enum to smooth the creation of browsers of different types. First the enum:

enum BrowserKind
{
    InternetExplorer,
    IE = InternetExplorer,
    Firefox,
    Chrome,
    PhantomJS
}

Now, let's create the factory method which instantiates the browsers. I'm not showing the class declaration to save space, but I'm creating the factory methods in a static class called WebDriverFactory.

public static IWebDriver CreateWebDriverWithProxy(BrowserKind kind,
                                                  Proxy proxy)
{
    IWebDriver driver = null;
    switch (kind)
    {
        case BrowserKind.InternetExplorer:
            driver = CreateInternetExplorerDriverWithProxy(proxy);
            break;

        case BrowserKind.Firefox:
            driver = CreateFirefoxDriverWithProxy(proxy);
            break;

        case BrowserKind.Chrome:
            driver = CreateChromeDriverWithProxy(proxy);
            break;

        default:
            driver = CreatePhantomJSDriverWithProxy(proxy);
            break;
    }

    return driver;
}

Now, I'll list out each of the driver creation methods. These are pretty self-explanatory, but quirks of each driver are noted in the comments in the source code.

private static IWebDriver CreateInternetExplorerDriverWithProxy(Proxy proxy)
{
    InternetExplorerOptions ieOptions = new InternetExplorerOptions();
    ieOptions.Proxy = proxy;

    // Make IE not use the system proxy, and clear its cache before
    // launch. This makes the behavior of IE consistent with other
    // browsers' behavior.
    ieOptions.UsePerProcessProxy = true;
    ieOptions.EnsureCleanSession = true;

    IWebDriver driver = new InternetExplorerDriver(ieOptions);
    return driver;
}

private static IWebDriver CreateFirefoxDriverWithProxy(Proxy proxy)
{
    // A future version of the .NET Firefox driver will likely move
    // to an "Options" model to be more consistent with other browsers'
    // API.
    FirefoxProfile profile = new FirefoxProfile();
    profile.SetProxyPreferences(proxy);

    IWebDriver driver = new FirefoxDriver(profile);
    return driver;
}

private static IWebDriver CreateChromeDriverWithProxy(Proxy proxy)
{
    ChromeOptions chromeOptions = new ChromeOptions();
    chromeOptions.Proxy = proxy;

    IWebDriver driver = new ChromeDriver(chromeOptions);
    return driver;
}

private static IWebDriver CreatePhantomJSDriverWithProxy(Proxy proxy)
{
    // This is an egregiously inconsistent API. Expect this to change
    // so that an actual Proxy object can be passed in.
    PhantomJSDriverService service =
        PhantomJSDriverService.CreateDefaultService();
    service.ProxyType = "http";
    service.Proxy = proxy.HttpProxy;

    IWebDriver driver = new PhantomJSDriver(service);
    return driver;
}

Now that we have the WebDriverFactory class created, we can update our main method to its final form, which is the following:

static void Main(string[] args)
{
    // Note that we're using a desired port of 0, which tells
    // Fiddler to select a random available port to listen on.
    int proxyPort = StartFiddlerProxy(0);

    // We are only proxying HTTP traffic, but could just as easily
    // proxy HTTPS or FTP traffic.
    OpenQA.Selenium.Proxy proxy = new OpenQA.Selenium.Proxy();
    proxy.HttpProxy = string.Format("127.0.0.1:{0}", proxyPort);

    // You can uncomment any of the lines below to verify that the
    // retrieval of HTTP status codes works properly for each browser.
    IWebDriver driver = WebDriverFactory.CreateWebDriverWithProxy(BrowserKind.IE, proxy);
    //IWebDriver driver = WebDriverFactory.CreateWebDriverWithProxy(BrowserKind.Firefox, proxy);
    //IWebDriver driver = WebDriverFactory.CreateWebDriverWithProxy(BrowserKind.Chrome, proxy);
    //IWebDriver driver = WebDriverFactory.CreateWebDriverWithProxy(BrowserKind.PhantomJS, proxy);

    TestStatusCodes(driver);

    driver.Quit();

    StopFiddlerProxy();
    Console.WriteLine("Complete! Press <Enter> to exit.");
    Console.ReadLine();
}

We're pretty much done with our final solution, except for one final tweak. Let's revisit our NavigateTo and ClickNavigate methods from Part 2 which actually retrieve the HTTP status code. Take a look at the signatures of each of those methods:

public static int NavigateTo(IWebDriver driver, string targetUrl)
public static int ClickNavigate(IWebElement element)

One of the super-groovy things about the .NET Framework since version 3.0 is the introduction of extension methods. These allow you to extend a type with methods of your own design, allowing you to write code as if that type had that method to begin with. Our two methods are tailor-made to be used as extension methods. Simply changing the signature to the following will make that work. I'd also recommend moving those methods to a new static class named something like ExtensionMethods for clarity, but that's up to you.

public static int NavigateTo(this IWebDriver driver, string targetUrl)
public static int ClickNavigate(this IWebElement element)

That means that the final version of our TestStatusCodes method looks like this:

private static void TestStatusCodes(IWebDriver driver)
{
    // Using Mozilla's main page, because it demonstrates some of
    // the potential problems with HTTP status code retrieval, and
    // why there is not a one-size-fits-all approach to it.
    string url = "http://www.mozilla.org/";

    // Note that the standard IWebDriver interface doesn't have
    // a NavigateTo() method that takes a URL and returns a status
    // code. However, thanks to the magic of extension methods, 
    // we can make it look like it does, and call it directly off
    // the driver object.
    int responseCode = driver.NavigateTo(url);
    Console.WriteLine("Navigation to {0} returned response code {1}",
                      url, responseCode);

    string elementId = "firefox-promo-link";

    // We're using the same extension method magic here to add in
    // a ClickNavigate() method which looks like it's directly
    // implemented by IWebElement, even though it really isn't.
    IWebElement element = driver.FindElement(By.Id(elementId));
    responseCode = element.ClickNavigate();
    Console.WriteLine("Element click returned response code {0}",
                      responseCode);

    // Demonstrates navigating to a 404 page.
    url = "http://www.mozilla.org/en-US/doesnotexist.html";
    responseCode = driver.NavigateTo(url);
    Console.WriteLine("Navigation to {0} returned response code {1}",
                      url, responseCode);
}

We'd also probably want to revisit our timeout code in those methods, probably by providing additional overloads that would make it configurable. I've done that in my local version, and it seems to work pretty well. If you want to see all of this code in a single place, you can take a look at the GitHub repository for this and other example projects on using a proxy.

The argument of the WebDriver project committers regarding HTTP status codes is that a method to retrieve them is out of scope for the API. Furthermore, the explanation has been that the proper approach, one that will work for all browsers, without introducing a suboptimal feature to the WebDriver API, is to use a proxy to capture the HTTP traffic and analyze it yourself. The response to that argument has often been that's too hard to do, and it's stupid to use a screwdriver to put in a screw, when one has a hammer that will work just as well. Hopefully, with this series of blog posts, I've shown that it's pretty easy to work out the use of a proxy to get the information you want. My example is in the .NET bindings, but Java, Ruby, and Python examples would look similar, when using a software-based proxy written in those languages.