tag:blogger.com,1999:blog-74337395588629856362024-03-24T16:32:35.655-07:00Rantings of a Selenium ContributorJim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.comBlogger32125tag:blogger.com,1999:blog-7433739558862985636.post-5040815365127390832020-04-20T14:19:00.000-07:002020-04-20T14:19:58.365-07:00Setting Up a Windows Development Environment for the Selenium .NET Language Bindings<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
One of the most common challenges I come across with users trying to work on developing the .NET
language bindings is how to set up an environment on their machine to be able to build the language
bindings and get them to build. When I and other committers facilitate the "Fix a Bug, Become a
Committer" workshop at various Selenium Conferences, the vast majority of issues we see for people
is in setting up their development environment to get the build tools working. Over the last couple
of years, the Selenium project has put a fair amount of time and energy into unifying the build
system across language bindings, but this is especially challenging for Windows users, as most of
the tooling for open source software projects treat Windows as a second-class citizen. I've had to
build and rebuild development environments often enough that I understand all of the gotchas involved
with setting up such an environment, but I've never documented the process outside of my own head.
This post is an attempt to provide a more definitive set of steps for setting up such an environment
for folks interested in building the Selenium .NET bindings on their own.<br />
<br />
</span>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
A Couple of Preliminary Notes
</span>
</h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
First, these instructions will be customized to use Windows 10. They may work on other versions
of Windows, but you should not expect the screenshots included later in this document to exactly
match up with your experience.<br />
<br />
Second, we will be running several commands from the command line. This seems like it should be
table stakes for most open source developers, but there are many, many effective developers who
rely solely on graphical tools to do their work, especially on Windows. If you are not comfortable
with opening a command prompt and executing commands within it, you will need to become more
comfortable with it working with the Selenium code base.<br />
<br />
Third, <i><b>you will need administrative privileges on your machine to configure the environment.</b></i>
Unfortunately, there is no way around this. I recognize this is a barrier to entry for some people,
but a properly configured environment requires several tools that would not be on a typical IT
department's install list.<br />
<br />
Finally, to install the tooling in this document, we will be using Chocolatey (<a href="https://chocolatey.org/">https://chocolatey.org/</a>).
It's possible to accomplish all of these installs manually, and if you want to do this, you're
welcome to, but you'll need to make sure everything happens post-install, like environment variables
and paths getting updated to enable things to run from a command line. Installing the tooling
manually will be outside the scope of this document.<br />
<br />
</span>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Setting Up Windows Features
</span>
</h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
You will need to have certain features enabled in Windows 10 to be able to successfully build
Selenium code. Specifically, you will need to turn on "Developer Mode." To do this, you can open
the Windows 10 Settings app (click the gear icon on the Start Menu), and choosing "Update
and Security."<br />
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjUY1ConsUIwvD-U2eWunuYB7-htbNpKWOvbugog6Cv1T2EE3CFbdzaUQoQaTWmNKRFIeoqR9TBwjTnKDOYPlFLsuxqsfIIfxkwiPeouw8FXSXn1-R-sdGA06_XnCU_NCeRPefF0O3uvA/s1600/Settings+-+Update+and+Security.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1197" data-original-width="1600" height="239" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjUY1ConsUIwvD-U2eWunuYB7-htbNpKWOvbugog6Cv1T2EE3CFbdzaUQoQaTWmNKRFIeoqR9TBwjTnKDOYPlFLsuxqsfIIfxkwiPeouw8FXSXn1-R-sdGA06_XnCU_NCeRPefF0O3uvA/s320/Settings+-+Update+and+Security.png" width="320" /></a>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
In the Update and Security section, choose "For developers," and turn on "Developer mode." Windows
will install the feature for Developer mode.<br />
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhA240aAdsomtxtLatZgWkVFAanObdtc55FBXWHbxuD3Y9oafDx_wYMsom9w-le1oxaEvpc5XRvGOBXj4PHMWKGQBF8CnSt47RTNufb2mamOkZmojEkS21V-mVMAk4nVJGN6ad_xQzJeqQ/s1600/Settings+-+Developer+Mode.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1192" data-original-width="1600" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhA240aAdsomtxtLatZgWkVFAanObdtc55FBXWHbxuD3Y9oafDx_wYMsom9w-le1oxaEvpc5XRvGOBXj4PHMWKGQBF8CnSt47RTNufb2mamOkZmojEkS21V-mVMAk4nVJGN6ad_xQzJeqQ/s320/Settings+-+Developer+Mode.png" width="320" /></a>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
</span>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Installing the Tools
</span>
</h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
To work in the Selenium .NET code base on Windows, we are going to need the following tools:</span>
<br />
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">
Chocolatey, which we will use for installation of other tools.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">
Git, which will be used for getting and updating the Selenium source code</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">
A Java Development Kit (JDK) which will be used to build some components needed for testing the
.NET bindings. Note that a JDK is different from a Java Runtime Environment (JRE) which is what
most people install when they "install Java," and we will be using OpenJDK, version 11.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">
Python, which some build rules used by the Selenium build process requires. We will install the
latest Python 3.x version.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">
Visual Studio 2019, for modifying and building the C# code that make up the .NET bindings. It
is perfectly acceptable to install the Community Edition to work with the code base.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">
MSYS2, which is an alternative shell environment that provides Unix-like commands. While the
normal build process using Visual Studio does not require this, the command-line cross-language,
cross-platform build tool used to produce the .NET bindings assemblies does require it.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">
Bazel, the command-line build tool used to build components of the Selenium project in multiple
supported languages, including C#.</span></li>
</ul>
<br />
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Install Chocolatey
</span>
</h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
To install Chocolatey, you'll need to run an install script from within PowerShell, an enhanced
command line available in Windows. We also need to run PowerShell as an administrator so that the
subsequent packages being installed can be run without further elevation prompts. To open PowerShell
as an administrator, the easiest thing to do is search for it using the Windows Start Menu search
feature, and choose the option to "Run as Administrator."<br />
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWcZNNJesFnHPVLSi5uVc8V_4NgcoRNfpzVE01CKEeAAJHdyUXdHmTDYIwTUrQdG-QseDyq8MyKwtLO07upA3OJQfaoaPr4MGiiLZx7lu38g4MZMyvU3cnh2QvXl_JXGb-MefbX1kekq8/s1600/Start+Menu+-+Search+-+PowerShell.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1324" data-original-width="1600" height="264" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWcZNNJesFnHPVLSi5uVc8V_4NgcoRNfpzVE01CKEeAAJHdyUXdHmTDYIwTUrQdG-QseDyq8MyKwtLO07upA3OJQfaoaPr4MGiiLZx7lu38g4MZMyvU3cnh2QvXl_JXGb-MefbX1kekq8/s320/Start+Menu+-+Search+-+PowerShell.png" width="320" /></a>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
In the resulting PowerShell window, type the following command:<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
Set-ExecutionPolicy Bypass -Scope Process -Force;
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072;
iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
When this is done, you should be able to type <span style="font-family: "courier new" , "courier" , monospace;">choco</span>
into the PowerShell command prompt and receive an informational message that includes the version
of Chocolatey installed.<br />
<br />
</span>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Install Git
</span>
</h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
The Selenium project uses Git (<a href="https://git-scm.com/">https://git-scm.com/</a>) as its source
control system. To get the source code on your machine and to keep it updated, you will need to have
Git installed. To install it using Chocolatey, type the following command in the PowerShell window,
and answer the prompts:<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
choco install git<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
When this command completes, you should be able to open a regular command prompt
and type <span style="font-family: "courier new" , "courier" , monospace;">git</span>,
and receive an information message about the usage of Git.<br />
<br />
</span>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Install OpenJDK 11
</span>
</h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
The .NET language bindings of the Selenium project share the same pages for testing that the Java
language bindings do. To enable the running of the .NET bindings' test suite, we need to be able to
build the Java language bindings as well, so that we can build the web server that the test suite
uses to serve the pages for the browser to browse to during the tests. The Selenium project requires
Java 11 to build the Java components, and the OpenJDK is preferred. To install OpenJDK 11 using
Chocolatey, type the following command in the PowerShell window, and answer the prompts.<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
choco install openjdk11<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
When this command completes, you should be able to open a regular command prompt and type
<span style="font-family: "courier new" , "courier" , monospace;">javac</span>,
and receive an information message about the use of the Java compiler. If you can type
<span style="font-family: "courier new" , "courier" , monospace;">java</span>
and receive an informational message, but typing <span style="font-family: "courier new" , "courier" , monospace;">javac</span>
yields a "command not recognized" error, you've installed a Java Runtime Environment (JRE),
and not a Java Development Kit (JDK). You must have a JDK.<br />
<br />
</span>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Install Python
</span>
</h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Some of the Bazel rules used by the Selenium project require Python (<a href="https://www.python.org">https://www.python.org/</a>)
to execute. To install Python using Chocolatey, type the following command in the
PowerShell window, and answer the prompts:<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
choco install python<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
When this command completes, you should be able to open a regular command prompt and
type <span style="font-family: "courier new" , "courier" , monospace;">python</span>,
and enter the Python interactive debugging environment (also called a REPL). To exit
the REPL, enter <span style="font-family: "courier new" , "courier" , monospace;">quit()</span>.<br />
<br />
</span>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Install Visual Studio 2019 Community Edition
</span>
</h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
The Selenium project uses Bazel as its cross-language build tool, which means that to build the
.NET bindings, you need the prerequisites that Bazel requires to build C# code, including Visual
Studio 2019, and the project will build successfully using the free Community Edition
(<a href="https://visualstudio.microsoft.com/downloads/">https://visualstudio.microsoft.com/downloads/</a>).
Among those prerequisites are a C++ compiler. Even though you may not be building any C++ code as
part of the Selenium project, Bazel still requires access to a C++ compiler to build tools to compile
other language elements. Additionally, after you install Visual Studio, you may need to add optional
components that are not automatically selected during the Visual Studio install. To install Visual
Studio 2019 Community Edition using Chocolatey, type the following command in the PowerShell window
and answer the prompts.<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
choco install visualstudio2019community<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
When the command completes, you should have a working installation of Visual Studio on your machine.
You may need to reboot to fully complete the installation. After installing Visual Studio, you will
need to make sure the correct optional components are installed as well. To do this, launch the
Visual Studio Installer from the Windows Start Menu.<br />
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhnLxxGuKSj0vT4KXS5r0HiOxtrS-Vsq0L6FZ2PihhMlXUsfgZtJVvYEuWg_USghETsZM6ESI_WwrHMLkMLMdC2kMj43gvo1x7rtAVnCRhoMcgRWI4fqSU2pHbHQht9Odvw06mh6qWD8Q/s1600/VS+Installer+-+start+menu.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1396" data-original-width="736" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhnLxxGuKSj0vT4KXS5r0HiOxtrS-Vsq0L6FZ2PihhMlXUsfgZtJVvYEuWg_USghETsZM6ESI_WwrHMLkMLMdC2kMj43gvo1x7rtAVnCRhoMcgRWI4fqSU2pHbHQht9Odvw06mh6qWD8Q/s320/VS+Installer+-+start+menu.png" width="168" /></a>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
Once the installer launches, click the "Modify" button.<br />
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAI0N6zZUPNbYTjwG3QTlohaojz1tlztMJzi5x6tw3NBbcSfZFylj5Ox8tDzRK81kBgs-qsGhFNdumH4JDl3H2ZCRUkwfKpuTS77WZbmp7ZzBKdUDCEvtkbaiN7V4M70NGIhqeUdmgUxw/s1600/VS+Installer+-+Modify.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAI0N6zZUPNbYTjwG3QTlohaojz1tlztMJzi5x6tw3NBbcSfZFylj5Ox8tDzRK81kBgs-qsGhFNdumH4JDl3H2ZCRUkwfKpuTS77WZbmp7ZzBKdUDCEvtkbaiN7V4M70NGIhqeUdmgUxw/s320/VS+Installer+-+Modify.png" width="320" height="180" data-original-width="1600" data-original-height="900" /></a>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
In the ensuing component selection dialog, make sure that the ".NET desktop development" workflow
is installed, and that the following optional components are installed as part of that workflow:<br />
<br />
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">.NET Core development</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">.NET Core 2.1 LTS Runtime</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">.NET Framework 4-4.6 development tools</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">.NET Framework 4.6.1 development tools</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">.NET Framework 4.6.2 development tools</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">.NET Framework 4.7 development tools</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">.NET Framework 4.7.1 development tools</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">.NET Framework 4.8 development tools</span></li>
</ul>
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLsYRd9HoCTJgHwl8B1A4kTpsm0Sq2T82CAHvKwJU630Tk_BLA7OPLXcR57vTXSFxjCxn3lpc3u36z1kqCTMYXPLT8T9qn6iWArjByHDwJ0FLQ-0RP3anU-NIGBtetOL7qUb4gsursEpM/s1600/VS+Installer+-+C%2523+options.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLsYRd9HoCTJgHwl8B1A4kTpsm0Sq2T82CAHvKwJU630Tk_BLA7OPLXcR57vTXSFxjCxn3lpc3u36z1kqCTMYXPLT8T9qn6iWArjByHDwJ0FLQ-0RP3anU-NIGBtetOL7qUb4gsursEpM/s320/VS+Installer+-+C%2523+options.png" width="320" height="162" data-original-width="1600" data-original-height="811" /></a>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
You will also need to make sure the "Desktop development with C++" workflow is installed,
and then you can click the "Modify" button to install the optional components.<br />
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMnTyWtK5dlocmdEFqR3qTurSPO30-cWyl3TQcgTQEHsZslp0Z81A0rifw-CWU4_14pM9nmzScYjX3UB6QWRl_5WcMGW5aWPag1Jbh0vm9WxKc5zsmEgY4zS7Ry80d3SnW9BGL7-UMH7E/s1600/VS+INstaller+-+C%252B%252B+options.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMnTyWtK5dlocmdEFqR3qTurSPO30-cWyl3TQcgTQEHsZslp0Z81A0rifw-CWU4_14pM9nmzScYjX3UB6QWRl_5WcMGW5aWPag1Jbh0vm9WxKc5zsmEgY4zS7Ry80d3SnW9BGL7-UMH7E/s320/VS+INstaller+-+C%252B%252B+options.png" width="320" height="162" data-original-width="1600" data-original-height="811" /></a>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
Once the installation is complete, you can launch Visual Studio to make sure the IDE runs properly.<br />
<br />
</span>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Install MSYS2
</span>
</h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Some of the build rules that the Bazel build tool uses in the Selenium build process are not completely
ported to work seamlessly with Windows, and hard-code the use of Unix-like shell commands. To support
this as part of the build process, Bazel will require the installation of the MSYS2 subsystem
(<a href="https://www.msys2.org/">https://www.msys2.org/</a>). Because of other configuration requirements,
it is important to know the directory where the MSYS2 system is installed. This example will use C:\tools\msys64
as the install location, but you can substitute another location if you wish. To install MSYS2 to a specific
location using Chocolatey, type the following command in the PowerShell window and answer the prompts.<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
choco install msys2 --params "/InstallDir=C:\tools\msys64"<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
Once the installation is completed, you should see the MSYS2 tools installed at the location you specified.
So that Bazel can also find those tools at the specified location, you will need to set up an environment
variable for Bazel to use. To accomplish this, the easiest thing to do is search for the environment
variable editor using the Windows Start Menu search feature, and choose the option to "Open."<br />
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjAjRt6t3macTR82fkwPe5gFLtu0PshH6Gy9LFbG1kUtvS59mxB5UEkkyA3rlRG50mkmJWHlAr4UxlnQm_4PqhxhoxOvYI942JEYjop8aprJfQGWZaauy_ymhLFkUmZ4devxf15w4Djf4/s1600/Start+Menu+-+Search+-+Edit+System+Environment+Variables.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1310" data-original-width="1600" height="262" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjAjRt6t3macTR82fkwPe5gFLtu0PshH6Gy9LFbG1kUtvS59mxB5UEkkyA3rlRG50mkmJWHlAr4UxlnQm_4PqhxhoxOvYI942JEYjop8aprJfQGWZaauy_ymhLFkUmZ4devxf15w4Djf4/s320/Start+Menu+-+Search+-+Edit+System+Environment+Variables.png" width="320" /></a>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
In the Environment Variables dialog, click the "New..." button under System variables.<br />
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRkns2W8w6kTgYgwevka2iz_55RPHc57nwFMXxl_xuKE9uwH5f9_DPDfLcCfVztt6SwBwe1fJ940olDAMr5etTVNysx8FNFRIckmTW06z5n2gBllHOxjob1ZeodAultrSVvyQ_-g-o-2Q/s1600/Enfironment+Variables.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1170" data-original-width="1240" height="301" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRkns2W8w6kTgYgwevka2iz_55RPHc57nwFMXxl_xuKE9uwH5f9_DPDfLcCfVztt6SwBwe1fJ940olDAMr5etTVNysx8FNFRIckmTW06z5n2gBllHOxjob1ZeodAultrSVvyQ_-g-o-2Q/s320/Enfironment+Variables.png" width="320" /></a>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
In the New System Variable dialog, enter <span style="font-family: "courier new" , "courier" , monospace;">BAZEL_SH</span>
as the variable name, and the path to <span style="font-family: "courier new" , "courier" , monospace;">bash.exe</span>
in your MSYS2 installation as the variable value. The <span style="font-family: "courier new" , "courier" , monospace;">bash.exe</span> executable will be located in the <span style="font-family: "courier new" , "courier" , monospace;">usr\bin</span>
directory of your MSYS2 installation. Using our example path for installation, the path to add would
be <span style="font-family: "courier new" , "courier" , monospace;">C:\tools\msys64\usr\bin\bash.exe</span>.
If you chose a different installation path, your path to enter would be different.<br />
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtew29kbK3go8PWTcMECu4eWKeYkq_0RJe-oD6QwhtD7P5eJbiJFWv0_cqIlV_BkCHjZ-UqtuxHQnGnNmDa5WC3Xb629Hvz5YUNrXxok1gl_ihvUG99YDx0mAcmZ045WBKIak4-SWFegs/s1600/New+Environment+Variable.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtew29kbK3go8PWTcMECu4eWKeYkq_0RJe-oD6QwhtD7P5eJbiJFWv0_cqIlV_BkCHjZ-UqtuxHQnGnNmDa5WC3Xb629Hvz5YUNrXxok1gl_ihvUG99YDx0mAcmZ045WBKIak4-SWFegs/s320/New+Environment+Variable.png" width="320" height="82" data-original-width="1306" data-original-height="336" /></a>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
Once you have entered those values, click OK on the New System Variable and the Environment Variables dialogs.<br />
<br />
</span>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Install Bazel
</span>
</h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
You can now install the final tool in the development environment tool chain, Bazel
(<a href="https://bazel.build/">https://bazel.build/</a>). Bazel is a cross-language build environment
that creates repeatable builds and uses caching to avoid building components unnecessarily, dramatically
decreasing build times. To install Bazel using Chocolatey, type the following command in the PowerShell
window and answer the prompts.<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
choco install bazel<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
When this command completes, you should be able to open a regular command prompt and type
<span style="font-family: "courier new" , "courier" , monospace;">bazel</span>, and
receive an informational message about the use of the command.<br />
<br />
</span>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Getting and Building Selenium Code
</span>
</h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
After running all of the installers for getting the build tools, it's probably a good idea to
reboot your machine so that registry changes from the installers can take effect. Once the reboot
is complete, and now that we have all of the tooling in place to build Selenium artifacts, you can
fetch the code from the source control repository on GitHub. To make and submit changes to the
project, you will need to fork the project on GitHub, and submit pull requests to the project.
Performing those tasks is beyond the scope of this document. To get the Selenium code base copied
to your machine, open a Developer Command Prompt for Visual Studio 2019 from the Windows Start Menu.<br />
<br />
</span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0Dn1U0J9OREu6PyX_9UwjPwL2c3BKulnN9vOYHzHCSJMWq9hPk5QXOMaAAcS772lY-d0HHV96o9-GXOk_uY2xkYw785lF6xaGlMWhyphenhyphenF3h4kB3JzT2gdiPFkwxbom41UQ6sjEpZ-lD2KE/s1600/Developer+Command+Prompt+for+VS+2019.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1432" data-original-width="804" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0Dn1U0J9OREu6PyX_9UwjPwL2c3BKulnN9vOYHzHCSJMWq9hPk5QXOMaAAcS772lY-d0HHV96o9-GXOk_uY2xkYw785lF6xaGlMWhyphenhyphenF3h4kB3JzT2gdiPFkwxbom41UQ6sjEpZ-lD2KE/s320/Developer+Command+Prompt+for+VS+2019.png" width="179" /></a></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
In the resulting command prompt window, navigate to an appropriate directory, and clone the project.
I usually keep my projects in a directory called C:\Projects, but you can choose whatever directory
you like. If you use a different directory structure, keep in mind that you'll need to use those
paths when entering commands. To clone the project in your desired location, enter the following
commands in your command prompt.<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
cd \Projects<br />
git clone https://github.com/SeleniumHQ/selenium<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
A quick warning, the Selenium Git repository is very, very large. This is due to the use of a monorepo,
where dependencies are kept checked into the source tree so that anyone can build without the requirement
of a network connection to download dependencies. If you don't need the full source control history, or
are in a hurry, you can add the <span style="font-family: "courier new" , "courier" , monospace;">--depth=1</span>
switch to the clone command (e.g., <span style="font-family: "courier new" , "courier" , monospace;">
git clone --depth=1 https://github.com/SeleniumHQ/selenium</span>) to only get the tip of the tree and build from there.<br />
<br />
When this command finishes, the repo should be cloned to your local machine in the
<span style="font-family: "courier new" , "courier" , monospace;">selenium</span>
subdirectory. To make sure that you can build from the command line, enter the following commands
in your command prompt window.<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
cd selenium<br />
bazel build //dotnet/test/common:chrome<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
This will build everything the .NET language bindings depend on, including the JavaScript automation atoms,
the Java test web server, and the .NET bindings assemblies as well as the NUnit test assemblies, but will
stop just short of running the tests. Be aware that this will take awhile the first time you attempt it.
Many of the tasks will not require rebuilds once built for the first time.<br />
<br />
</span>
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Summary
</span>
</h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
At this point, you should be able to successfully build the .NET bindings from the command line using Bazel.
You should also be able to open the <span style="font-family: "courier new" , "courier" , monospace;">WebDriver.NET.sln</span>
file in the project's source tree and modify, build, and test using Visual Studio. As a reminder, here
is a summary of the steps, assuming you are using the paths that are listed in the preceding text;
your paths may need to be different if you've chosen different locations.<br />
<br />
In an elevated PowerShell command window, run the following commands:<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
Set-ExecutionPolicy Bypass -Scope Process -Force;
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072;
iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))<br />
choco install git<br />
choco install openjdk11<br />
choco install python<br />
choco install visualstudio2019community<br />
choco install msys2 --params "/InstallDir=C:\tools\msys64"<br />
choco install bazel<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
Update the Visual Studio installation through the installer to include the ".NET desktop development"
and "Desktop development with C++" workflows with the appropriate optional components, and add the
<span style="font-family: "courier new" , "courier" , monospace;">BAZEL_SH</span>
environment variable to point to the MSYS2 Bash shell executable. After installing these packages,
you'll probably want to reboot your machine.<br />
<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
In a Developer Command Prompt for Visual Studio 2019, run the following commands:<br />
<br />
</span>
<span style="font-family: "courier new" , "courier" , monospace;">
cd \Projects<br />
git clone https://github.com/SeleniumHQ/selenium<br />
cd selenium<br />
bazel build //dotnet/test/common:chrome<br />
</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<br />
To run the .NET tests from the command line, a few additional configuration options are required,
but we will cover that in a separate blog post. If you run into issues with these steps, feel
free to contact me on the Selenium IRC (#selenium at freenode.net) or Slack channels.
</span>
</div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com0tag:blogger.com,1999:blog-7433739558862985636.post-58571837760320416082019-07-08T16:15:00.000-07:002019-07-08T16:15:10.204-07:00Announcing Selenium .NET Bindings 4.0 alpha 2<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">I'm very proud to announce the 4.0-alpha2 release of the Selenium .NET bindings! There are several exciting things to look forward to in this release. The first is the fixing of several issues that have cropped up because beginning with version 75, Chrome (and ChromeDriver) now use the <a href="https://w3c.github.io/webdriver/">W3C WebDriver Specification</a> as the default protocol dialect for communication between Selenium and the driver. This has led to a few issues like the <span style="font-family: "Courier New", Courier, monospace;">loggingPrefs</span> capability being renamed, and the legacy logging APIs (<span style="font-family: "Courier New", Courier, monospace;">driver.Manage().Logs</span>) no longer working. This functionality should be restored in this release.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">By far, the biggest addition to this release of the .NET bindings is the addition of integration with the Chrome DevTools Protocol (CDP). We are extremely excited to be able to bring this feature to users of Selenium. It allows users, using their existing WebDriver instance, to instantiate and use a CDP session, including the two-way communication of events. The .NET API for using CDP is still experimental, and may change between releases until the alpha/beta period is over. But to whet your appetite, here's what code to use this looks like in 4.0.0-alpha02:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<script src="https://gist.github.com/jimevans/6cc5aa733dda4cdf528f011ea28ab9f0.js"></script><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The API uses the .NET System.Net.WebSockets.ClientWebSocket implementation for communication with Chrome, which means it's limited to use with Windows 8.1 and above. This is a limitation of the WebSocket implementation, so complaints about that should be directed toward Microsoft. Accordingly, most of the CDP API is async, though the remainder of the WebDriver API is not.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Also, for the moment, the .NET bindings do not implement <a href="https://chromedevtools.github.io/devtools-protocol/">domains marked as "experimental" in the protocol definition</a>. One thing that we really do <i>not</i> want for Selenium is for it to be tied down to specific versions of Chrome. Since the DevTools Protocol is not subject to any standard, and can be changed at the whim of the Chromium developers, this seems like a potentially suboptimal solution if we do that.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">The CDP integration is something we'd really like to get feedback on, so give it a whirl, and let us know what you think. </span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com0tag:blogger.com,1999:blog-7433739558862985636.post-69166643805860721322019-06-04T14:08:00.000-07:002019-07-08T16:15:00.583-07:00Handling Authentication Requests with Selenium: Epilogue<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">I hope the preceding series of posts has been useful. To wrap things up, I want to share a <a href="https://github.com/jimevans/selenium-auth-with-proxy-examples">GitHub repository</a> that contains sample code for each of the items we've discussed. It includes an ASP.NET Core demo web site that implements Basic, Digest, and NTLM authentication. It includes sample <a href="https://www.seleniumhq.org/">Selenium</a> code using <a href="https://github.com/jimevans/BenderProxy">BenderProxy</a> (version 1.1.2 or later) and <a href="https://github.com/jimevans/passedball">PassedBall</a> (version 1.2.0 or later) to automate the site. The Selenium code runs in a console application, which will await you pressing the Enter key before shutting down the proxy and quitting the browser. This will allow you to see the state of the browser before everything quits. Other features of the sample repo include working factory classes for Selenium sessions and the demo cases themselves.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">To make the demo in the source repo properly work, <b><i>you must run it on Windows</i></b>, because we are enabling NTLM authentication. Also, <i><b>you will need administrative access on your Windows machine</b></i>, which is unfortunate, but there is no other way to get the development web server to listen on a host name other than "localhost". If you change the test to navigate to the site on "localhost", the browser will likely bypass the proxy, because most browsers bypass proxies browsing on localhost without taking other configuration steps. By default, the demo project uses <span style="font-family: "courier new" , "courier" , monospace;">www.seleniumhq-test.test</span> and port <span style="font-family: "courier new" , "courier" , monospace;">5000</span>, but you can use whatever you want. Here's how to configure your test environment so that the demo app will work properly:</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">From an elevated ("Run as Administrator") command prompt, edit your hosts file to contain a mapped entry for the host you wish to use. The hosts file can be edited in any text editor, including Notepad, so the following command will open it:</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">notepad.exe %WinDir%\System32\drivers\etc\hosts</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Once open, add the following line:</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">127.0.0.1 <host name></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Be sure to substitute your preferred host name for <span style="font-family: "courier new" , "courier" , monospace;"><host name></span> . Save and close the hosts file. As an aside, this is a very useful technique for Selenium code to simulate navigation to external sites without actually having to navigate outside one's local machine.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Also in the elevated command prompt, execute the following command:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">netsh http add urlacl url="http://<host name>:<port>/" user=everyone</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Be sure to substitute your preferred host name and port for <span style="font-family: "courier new" , "courier" , monospace;"><host name></span> and <span style="font-family: "courier new" , "courier" , monospace;"><port></span> respectively. You should see a message that the URL reservation was successfully added. Now, this is a dangerous command, because it does open up a URL reservation for everyone, so you don't want to leave this permanently in place. You can remove it at any time after you're done using the sample by using another elevated command prompt to execute:</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">netsh http remove urlacl url="http://<host name>:<port>/"</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Once you've added the hosts file entry and the URL ACL, you're ready to load and run the authentication tests. Open the solution in Visual Studio 2019, and you should be able to build and run. When running, the solution runs a console application that will launch the test web app, start the proxy server, start a browser configured to use the proxy with Selenium, navigate to a protected URL for a specific authentication scheme, and then wait for the Enter key to be pressed. This will let you examine the browser to validate that, yes, the authentication succeeded You can also examine the diagnostic output written to the console by the test code, which describes the <span style="font-family: "courier new" , "courier" , monospace;">WWW-Authenticate</span> and <span style="font-family: "courier new" , "courier" , monospace;">Authorization</span> headers being used. Once you've validated to your satisfaction that in the browser really did authenticate using Selenium and without prompting the user, you can press Enter, which will quit the browser, stop the proxy server, and shut down the test web app. As an extra validation step, you can also start the test web app from Visual Studio and manually navigate to the URLs to validate that they really do prompt for credentials when browsed to.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Here's the Main method of the test app:</span><br />
<br />
<script src="https://gist.github.com/jimevans/73fd2debe9427a89712c9d82bb411f91.js"></script>
<span style="font-family: "arial" , "helvetica" , sans-serif;">As you can see, you can change the browser being used (line 5 in the listing above), and the authentication type (line 9 in the listing above) being tested by changing the commented lines in the main method. If you decided to use a different host name or port, you can also change that by uncommenting and changing the appropriate lines (lines 15 and 16 in the listing above, respectively).</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Hopefully, this series has given you some insights into how browsers perform authentication, and how it's possible to automate this using Selenium, without resorting to other UI automation tools. Happy coding!</span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com0tag:blogger.com,1999:blog-7433739558862985636.post-73919184365702993662019-06-03T06:41:00.000-07:002020-02-21T13:05:39.444-08:00Handling Authentication Requests with Selenium - Part 4: NTLM Authentication<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Now that we've created a thorough intellectual framework for how to handle authentication requests using Selenium in combination with a web proxy, and thanks to our <a href="https://jimevansmusic.blogspot.com/2019/05/handling-authentication-requests-part3.html">last post</a>, we can handle more than Basic authentication, let's take things a step further, and see how you can use Selenium in automating pages secured with NTLM authentication. Before we can do that, though, we need to have an understanding of how NTLM authentication differs from the previous types of authentication we've used before.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">NTLM authentication is a Microsoft-developed technology, originally implemented in the company's IIS web server product. It's not widely used on the public internet, but it does integrate nicely with things like Active Directory, so it can be quite useful for web applications used on company intranets that require security based on Active Directory credentials. This means that to provide sample code, we'll need to have a few things in place first. First, we'll need a test website that we can run locally, running on a server that implements NTLM authentication. Since we're working in C# in this series, we can create an ASP.NET Core web project to do that.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Second, we'll also need to host the application using Windows. Even though the ASP.NET Core project can run against .NET Core, and that can run on platforms other than Windows, we'll need to actually run on Windows to take advantage of NTLM authentication, unless we want to introduce a ton of complexity with Active Directory domains and the like (which we don't for this post).</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Finally, most browsers bypass the use of a proxy when running strictly on localhost. This means that if you're running things all on the same system, you'll need to either configure the browser not to do this, or trick it into thinking the site the browser is connecting to isn't the local machine. The latter is far easier, since it only involves adding a line to the Windows hosts file (located at </span><span style="font-family: "courier new" , "courier" , monospace;">%WinDir%\System32\drivers\etc\hosts</span><span style="font-family: "arial" , "helvetica" , sans-serif;">). On my test system, I've redirected <span style="font-family: "courier new" , "courier" , monospace;">www.seleniumhq-test.test</span> to <span style="font-family: "courier new" , "courier" , monospace;">127.0.0.1</span> by using the hosts file, and the sample code will reflect this.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">NTLM authentication is a challenge-response based authentication scheme, and it differs from other HTTP authentication schemes in that it authenticates a <i>connection</i>, not an individual request. This means that the browser and server must support so-called "keep-alive," or persistent TCP connections between them. It also means that our proxy has to support persistent TCP connections, and must allow us to use that exact connection for making the requests. Fortunately, the proxy we've been using so far, <a href="https://github.com/jimevans/BenderProxy">BenderProxy</a>, does support this.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The challenge-response mechanism used is complicated. <a href="https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-nlmp/b38c36ed-2804-4868-a9ff-8dd3182128e4">Very complicated</a>. So again, we'll be using the <a href="https://github.com/jimevans/passedball">PassedBall</a> library to parse authentication headers and generate authorization responses. It also requires multiple request/response round trips to perform the authentication handshake. Here's the implementation code for handling the NTLM authentication challenge for a sample site hosted on our local host machine:</span><br />
<br />
<script src="https://gist.github.com/jimevans/831b33d33c46dfd8c24aaf833407ea6e.js"></script>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Note carefully that the initial <span style="font-family: "courier new" , "courier" , monospace;">401 Unauthorized</span> response may contain multiple <span style="font-family: "courier new" , "courier" , monospace;">WWW-Authenticate</span> headers, so one may need to make sure the proper one is being used to interpret the response. Browsers, when faced with this, will usually choose what they perceive to be the "strongest" authentication method. In our case, we need to do that determination for ourselves.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">We'll wrap up this series with <a href="https://jimevansmusic.blogspot.com/2019/07/handling-authentication-requests-epilogue.html">one more post</a>, summing everything up.</span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com1tag:blogger.com,1999:blog-7433739558862985636.post-67499933868059069002019-05-31T05:22:00.000-07:002019-06-03T06:43:05.360-07:00Handling Authentication Requests with Selenium - Part 3: Beyond Basic Authentication<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In the <a href="https://jimevansmusic.blogspot.com/2019/05/handling-authentication-requests-part2.html">last post</a> in this series, we saw the general procedure for handling authentication requests with Selenium and a web proxy:</span><br />
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Start the programmable proxy</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Start a Selenium session configuring the browser to use the proxy</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Wire up a method to intercept the 401 Unauthorized response</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Use the method to resend the request with the correct Authorization header value</span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;">As we noted previously, the use of the Basic HTTP authentication scheme is rather weak. There are other authentication schemes that don't require the sending of a password in plain text over the wire. One such case is <a href="https://en.wikipedia.org/wiki/Digest_access_authentication">HTTP Digest authentication</a>. Let's see what that looks like. First, let's navigate to a page that implements Digest authentication, and examine what we see. As before, we'll use the hosted version of <a href="https://github.com/tourdedave/the-internet">The Internet</a> at <a href="http://the-internet.herokuapp.com/">http://the-internet.herokuapp.com/</a></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Browser sends:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">GET http://the-internet.herokuapp.com/digest_auth HTTP/1.1<br />
Host: the-internet.herokuapp.com<br />
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0<br />
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8<br />
Accept-Language: en-US,en;q=0.5<br />Accept-Encoding: gzip, deflate<br />
Connection: keep-alive<br />Upgrade-Insecure-Requests: 1</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Browser receives back:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">HTTP/1.1 401 Unauthorized <br />
Connection: keep-alive<br />
Content-Type: text/plain<br />
Content-Length: 0<br />
Www-Authenticate: Digest realm="Protected Area", nonce="MTU1ODkwNDI2MyBkYjYzMTA0ZTY0NmZjNmZhNDljNzQ2ZGY0ZTc3NDM4OA==", opaque="610a2ee688cda9e724885e23cd2cfdee", qop="auth"<br />
Server: WEBrick/1.3.1 (Ruby/2.2.5/2016-04-26)<br />
Date: Sun, 26 May 2019 20:57:43 GMT<br />
Via: 1.1 vegurTTP/1.1 200 OK </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br />
Note the value of the <span style="font-family: "courier new" , "courier" , monospace;">WWW-Authenticate</span> header, which is considerably more complex than in the Basic authentication scheme case. The algorithm for figuring out the correct value for the <span style="font-family: "courier new" , "courier" , monospace;">Authorization</span> header is likewise <a href="https://tools.ietf.org/html/rfc7616">much more complex</a>, which, in the simplest case, involves getting the MD5 hash of the string "userName:realm:password", then the MD5 hash of the HTTP verb and the URL of the resource being requested, then getting the Base64-encoded string of those two hashes along with the "nonce" value send in the authenticate header.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Whew. That's an awful lot to keep straight. Probably a little too complicated to post the code for resolving all of the nuances of it within this blog post. So it's time to introduce a new library to add to our toolbox for calculating the authorization header value for any of a variety of authentication methods. That library is called <a href="https://github.com/jimevans/passedball">PassedBall</a>, and it's available both on GitHub and as a NuGet package. Since PassedBall supports Digest authentication, and using the same process as in our previous post, here's the implementation of the method to intercept and resend the HTTP request:</span><br />
<br />
<script src="https://gist.github.com/jimevans/3ebf9a9bcae5c4ce952d71a4f2a721bf.js"></script>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Now that we have a library and generic framework for the generation of arbitrary authentication schemes, we'll look at <a href="https://jimevansmusic.blogspot.com/2019/06/handling-authentication-requests-part4.html">one last approach</a> for authentication, one that uses connection semantics for authentication, NTLM authentication.</span>
</div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com0tag:blogger.com,1999:blog-7433739558862985636.post-75900551199689612922019-05-30T07:24:00.001-07:002019-05-31T05:21:58.284-07:00Handling Authentication Requests with Selenium - Part 2: Using a Web Proxy for Basic Authentication<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">As I mentioned in the <a href="https://jimevansmusic.blogspot.com/2019/05/handling-authentication-requests-part1.html">immediately prior post in this series</a>, the way to avoid having the browser prompt for credentials while using a Selenium test is by supplying the correct information in the <span style="font-family: "courier new" , "courier" , monospace;">Authorization</span> header. Since Selenium's focus is automating the browser as close to how a user does so as possible, there's not a built-in way to examine or modify the headers. However, Selenium does make it very easy to configure the browser being automated to use a web proxy. A web proxy is a piece of software that stands between your browser and any request made of a web server, and can be made to examine, modify, or even block requests based on any number of rules. When configured to use a proxy, every request made by your browser flows through the proxy. Many businesses use proxies to ensure that only authorized resources are being accessed via business computers, or making sure that requests only come from authorized computers, or any number of other legitimate business purposes.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">How do you configure your browser to use a proxy with Selenium? The code looks something like this:</span><br />
<br />
<script src="https://gist.github.com/jimevans/ea03522d4f7645e331bbd1ec7d084148.js"></script>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Since we're Selenium users, we'll be using a proxy that allows us to programmatically start and stop it, and hook into the request/response chain via our code, and modify the results in order to interpret and replace the headers as needed. Any number of proxies could be used in this project. Many Selenium users have had great success using <a href="https://github.com/lightbody/browsermob-proxy">BrowserMob Proxy</a>, or there are commercial options like <a href="https://www.telerik.com/fiddler">Fiddler</a>. Since I personally prefer FOSS options, and don't want to leave the .NET ecosystem, for our examples here, we'll be using <a href="https://github.com/jimevans/BenderProxy">BenderProxy</a>. Here's the code for setting that up.</span><br />
<br />
<script src="https://gist.github.com/jimevans/00ff026d6078eb32802cf709308328b9.js"></script>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Now, how do we wire up the proper processing to mimic the browser's processing of an authentication prompt? We need to implement the addition of an <span style="font-family: "courier new" , "courier" , monospace;">Authorization</span> header that provides the correct value, for the authentication scheme requested by the server. BenderProxy's <span style="font-family: "courier new" , "courier" , monospace;">OnResponseReceived</span> handler happens after the response has been received from the web server, but before it's forwarded along to the browser for rendering. That gives us the opportunity to examine it, and resend another request with the proper credentials in the proper format. We're using the Basic authentication scheme in this example, and once again using <a href="http://the-internet.herokuapp.com/">The Internet</a> sample application. Here's the code for the method:</span><br />
<br />
<script src="https://gist.github.com/jimevans/4d35c88a22d89a6dcff9feb7fdb2debb.js"></script>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Running the code, we'll see that when the Selenium code is run, the browser will show the authorized page, as we intended. As you can tell from the implementation code, Basic authentication is pretty simple, sending the Base64 encoding of "userName:passsword". Its simplicity is also one reason it's not used very often, as it sends the credentials across the wire, essentially in clear text. There are other, more secure authentication schemes available, and they can be automated in similar ways. The trick is knowing how to specify the value for the <span style="font-family: "courier new" , "courier" , monospace;">Authentication</span> header. In the <a href="https://jimevansmusic.blogspot.com/2019/05/handling-authentication-requests-part3.html">next post</a> in the series, we'll look at another authentication mechanism, and how to handle something a little more complicated.</span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com2tag:blogger.com,1999:blog-7433739558862985636.post-61431757815856616852019-05-29T14:56:00.000-07:002019-05-30T07:31:34.249-07:00Handling Authentication Requests with Selenium - Part 1: How Does Browser Authentication Work Anyway?<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In order to understand how to use browser technologies to automate pages that use some form of authentication, it is useful to know what happens when you browse to such a page. What's actually happening when your browser prompts you for some form of credentials, usually a user name and password, before it will let you access a given resource on the web?</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">At the risk of dropping down to a ridiculously low level, let's talk about how browsers transfer data for browsing websites. First, an obligatory disclaimer. I'm going to deliberately gloss over using pages served via secure HTTP ("https"), and I'm going to ignore mostly-binary protocols like HTTP/2 for this series. Those items, while important, and may impact the outcomes you see here, are beyond the scope of this series.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Most of the time, a browser is using the Hypertext Transfer Protocol (or HTTP) to communicate with a given web server. When you type in a URL in your browser's address bar, your browser sends off an HTTP request (that's what the "http://" means at the beginning of the URL), and receives a response from the server. For the following examples, we'll be using Dave Haeffner's excellent Selenium-focused testing site, <a href="https://github.com/tourdedave/the-internet">The Internet</a>, which is designed to provide examples of challenging things a user might encounter when automating web pages with Selenium, and a hosted version of which is available at <a href="http://the-internet.herokuapp.com/">http://the-internet.herokuapp.com</a>. Here's what a typical exchange might look like:</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Browser sends:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">GET http://the-internet.herokuapp.com/checkboxes HTTP/1.1<br />
Host:the-internet.herokuapp.com<br />
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0<br />
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8<br />
Accept-Language:en-US,en;q=0.5<br />
Accept-Encoding:gzip, deflate<br />
Connection:keep-alive<br />
Upgrade-Insecure-Requests:1 </span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Browser receives back:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">HTTP/1.1 200 OK <br />
Connection: keep-alive<br />
Content-Type: text/html;charset=utf-8<br />
Content-Length: 2008<br />
Server: WEBrick/1.3.1 (Ruby/2.2.5/2016-04-26)<br />
Date: Thu, 23 May 2019 23:44:54 GMT<br />
Via: 1.1 vegur<br />
<br />
<body of HTML page here></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">This is what happens for virtually every time a browser makes a request for a resource. The important thing to note is in that first line of the response. The <span style="font-family: "courier new" , "courier" , monospace;">"200 OK"</span> bit means that the server had the resource and was sending it in response to the request. Now let's look at a request for a resource that is protected by authentication:</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Browser sends:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">GET http://the-internet.herokuapp.com/basic_auth HTTP/1.1<br />
Host: the-internet.herokuapp.com<br />
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0<br />
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8<br />
Accept-Language: en-US,en;q=0.5<br />
Accept-Encoding: gzip, deflate<br />
Connection: keep-alive<br />
Upgrade-Insecure-Requests: 1</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Browser receives back:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">HTTP/1.1 401 Unauthorized <br />
Connection: keep-alive<br />
Content-Type: text/html;charset=utf-8<br />
Www-Authenticate: Basic realm="Restricted Area"<br />
Content-Length: 15<br />
Server: WEBrick/1.3.1 (Ruby/2.2.5/2016-04-26)<br />
Date: Thu, 23 May 2019 23:52:24 GMT<br />
Via: 1.1 vegur</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Note the all-important first line of the response, which says <span style="font-family: "courier new" , "courier" , monospace;">"401 Unauthorized"</span>. That tells us that we have a page that requires authentication. Note that if you asked your browser to browse to the page <a href="http://the-internet.herokuapp.com/basic_auth">http://the-internet.herokuapp.com/basic_auth</a>, you would have been prompted for a user name and password. Note in the response the line that says <span style="font-family: "courier new" , "courier" , monospace;">Www-Authenticate: Basic realm="Restricted Area"</span>. That tells the browser that the "Basic" authentication scheme is expected, and that the user's user name and password are required, and so the browser prompts you, and then it re-sends the request to the server, but with an additional header. If you used the proper credentials for the aforementioned URL (user name: admin, password: admin), you'd see something like the following:</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Browser sends:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">GET http://the-internet.herokuapp.com/basic_auth HTTP/1.1<br />
Host: the-internet.herokuapp.com<br />
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0<br />
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8<br />
Accept-Language: en-US,en;q=0.5<br />
Accept-Encoding: gzip, deflate<br />
Connection: keep-alive<br />
Upgrade-Insecure-Requests: 1<br />
Authorization: Basic YWRtaW46YWRtaW4=</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Browser receives back:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">HTTP/1.1 200 OK <br />
Connection: keep-alive<br />
Content-Type: text/html;charset=utf-8<br />
Content-Length: 1643<br />
Server: WEBrick/1.3.1 (Ruby/2.2.5/2016-04-26)<br />
Date: Thu, 23 May 2019 23:59:31 GMT<br />
Via: 1.1 vegur<br />
<br />
<body of HTML page here></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Clearly, that additional header that says </span><span style="font-family: "courier new" , "courier" , monospace;">Authorization: Basic YWRtaW46YWRtaW4=</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> tells us that the browser must've done something with those credentials we gave it. If only we had a way to intercept the unauthorized response, calculate what needs to go into that authorization header, and resend the request before the browser had the chance to prompt us for credentials, we'd be golden. As luck (and technology) would have it, we do have exactly that ability, by using a web proxy. Every browser supports proxies, and Selenium makes it incredibly easy to use them with browsers being automated by it. The <a href="https://jimevansmusic.blogspot.com/2019/05/handling-authentication-requests-part2.html">next post in this series</a> will outline how to get that set up and working with Selenium.</span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com1tag:blogger.com,1999:blog-7433739558862985636.post-80892206948444572872019-05-29T02:32:00.000-07:002020-02-21T13:08:11.142-08:00Handling Authentication Requests with Selenium - Prologue<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">When you've been using Selenium for UI-based testing by using browser automation as long as I have, you occasionally see questions repeat themselves. Nearly any website of any complexity has functionality that relies on user authentication. Nowadays, that's most often done with some sort of form-based UI, which creates a session that gets tracked by the browser via a cookie. The variations of those are wide, but they don't require any additional tooling to handle via Selenium.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">However, every now and again, someone will post to one of the Selenium mailing lists or on the IRC/Slack channel asking about a site they have to automate that relies on the browser itself asking for the credentials with which to attempt authentication. Selenium is very good at automating pages within a browser, but makes little or no attempt at automating the parts of the browser outside the page being displayed, like file download selection dialogs, print dialogs, or, and most relevant here, browser-displayed credential dialogs. This usually means that the user tries to find another way to manipulate the dialog, and that often means a language-specific tool (like Java's <a href="https://docs.oracle.com/javase/7/docs/api/java/awt/Robot.html">Robot</a> class) or a platform-specific one (like <a href="https://www.autoitscript.com/site/">AutoIt</a>), or an approach that once worked with browser, but has now been deprecated and disallowed by nearly all as a security risk (putting the user name and password directly in the URL). The challenge with these approaches is that they rarely scale well, and almost never work correctly in the remote or grid case.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">This happens often enough that I'm going to start a series of posts about authentication, and how to effectively automate it with Selenium, without resorting to single-language or single-platform utilities. The last time I posted a series of blog posts about how to accomplish something using Selenium combined with other tools, <a href="https://github.com/SeleniumHQ/selenium-google-code-issue-archive/issues/141#issuecomment-191405372">one of the comments</a> I got was that they were "not impressed it [took] three blog articles to explain how to accomplish" the task.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The <a href="https://jimevansmusic.blogspot.com/2013/08/implementing-webdriver-http-status.html">blog post series</a> in question could easily have been done in one post, with a simple code example, but without any explanation of what the code was actually doing. I don't believe that approach is worthwhile, and is actually detrimental to learning.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">So let me lay this out ahead of time. The content of this post series could be done in a single post. It would be extraordinarily long, and a lot of it would end up saying, "go look at the sample code to get the full picture." I'd rather not take that approach. I'd much rather take my time, and at least attempt to give the relevant details in smaller, more quickly digested chunks. If you find that approach lacking, and would rather "just give me teh codez," I'll respectfully suggest heading elsewhere. As I add posts to the series, I'll try to keep an updated list of the parts of the series at the bottom of each post so that you can jump to the section you need.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">One last thing, I'll be showing code using C#. Assuming you have similar libraries available in your language of choice, the code can be ported to other languages and libraries. Doing so in this space will be beyond the scope of this post series. Also, I'll be using a more verbose coding style than is possible with modern versions of C#. This is a stylistic choice for explicitness and clarity; you're welcome to use more modern syntax in your own code.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Posts in this series:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://jimevansmusic.blogspot.com/2019/05/handling-authentication-requests-with.html">Prologue: This post</a></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://jimevansmusic.blogspot.com/2019/05/handling-authentication-requests-part1.html">Part 1: How Does Browser Authentication Work Anyway?</a></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://jimevansmusic.blogspot.com/2019/05/handling-authentication-requests-part2.html">Part 2: Using a Web Proxy for Basic Authentication</a></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://jimevansmusic.blogspot.com/2019/05/handling-authentication-requests-part3.html">Part 3: Beyond Basic Authentication</a> </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://jimevansmusic.blogspot.com/2019/06/handling-authentication-requests-part4.html">Part 4: NTLM Authentication</a> </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://jimevansmusic.blogspot.com/2019/07/handling-authentication-requests-epilogue.html">Epilogue: Final Thoughts</a> </span>
</div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com0tag:blogger.com,1999:blog-7433739558862985636.post-39876438370441154882019-04-19T00:06:00.000-07:002019-04-19T00:06:17.384-07:00Announcing Selenium 4.0 Alpha .NET Bindings<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">I am very proud to announce the release of the first alpha version of the Selenium 4.0 .NET language bindings! These bindings have been years in the making, and are now available for the first time in alpha form. They are by no means finished, and new features will be available before release. Some things to note about the bindings:</span><br />
<ul style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">The bindings now only support .NET Framework 4.5 and above, and .NET Core 2.0 and above (via .NET Standard). This is to gain support for additional classes in the .NET Framework that are unavailable in previous versions of the framework.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The internals of how the bindings communicate with the browser drivers has been completely rewritten to use <span style="font-family: "Courier New", Courier, monospace;">System.Net.Http.HttpClient</span>. I'm sure that something in this conversion has been missed, so there needs to be thorough testing of this.</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The bindings now only support the W3C WebDriver Specification dialect of the wire protocol. This simplifies the code for the .NET bindings considerably. </span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Methods and classes that were marked with the Obsolete attribute in 3.141 of the .NET bindings have been fully removed. This includes the <span style="font-family: "Courier New", Courier, monospace;">ExpectedConditions</span> and <span style="font-family: "Courier New", Courier, monospace;">PageFactory</span> classes. If you want to continue to use those structures, the <a href="https://github.com/DotNetSeleniumTools/DotNetSeleniumExtras">DotNetSeleniumExtras packages</a> will be updated for the final release. </span></li>
</ul>
<span style="font-family: Arial, Helvetica, sans-serif;">The complete list of changes is listed in the bindings' <a href="https://github.com/SeleniumHQ/selenium/blob/master/dotnet/CHANGELOG">CHANGELOG</a>. Please download and try out the bindings, and send your feedback. If you run into issues, you can file a new issue in the issue tracker at the <a href="https://github.com/SeleniumHQ/selenium">Selenium project GitHub repository</a></span><span style="font-family: Arial, Helvetica, sans-serif;"> or you can contact me via Twitter (@jimevansmusic) or on the Selenium project's IRC or Slack channel. Happy automating!</span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com0tag:blogger.com,1999:blog-7433739558862985636.post-59973574050198741702019-02-17T09:36:00.001-08:002019-02-17T09:36:32.758-08:00Improving IE Driver Use with Invalid Protected Mode Settings<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">Users using the IE driver when they do not have the ability to properly set the Protected Mode settings of the driver, usually restricted by an overzealous IT department, have always faced challenges when using the IE driver. This has been a known issue ever since the rewritten driver was introduced in 2011. The technical reasons for requiring the setting of Protected Mode settings are <a href="https://jimevansmusic.blogspot.com/2012/08/youre-doing-it-wrong-protected-mode-and.html">well-documented</a>, and haven't changed in the intervening years.<br /><br />In order to use the driver without setting the Protected Mode settings, the user had to resort to passing capabilities into the driver session creation, but this was still dicey because the driver could do nothing to mitigate when a Protected Mode boundary was crossed. So even then, it was possible, even likely, to receive errors like, "Unable to get current browser," or, "Unable to find element on closed window." As such, there was no conceivable way to work around the issue and still support all of the versions of Internet Explorer that were required. Since as of July 2019, the driver will support no versions other than IE 11, that landscape has changed somewhat. A <a href="https://github.com/SeleniumHQ/selenium/commit/91ea70d27b53d97538bc1de6690ce91691430e27">change to the IE driver</a> was recently (at the time of this writing) committed that makes the attempt to at least make the experience somewhat better.<br /><br />Now, when the user does not set the Protected Mode settings of the browser and sends the capability to bypass the checks for those settings, the driver will attempt to predict when a Protected Mode boundary will be crossed, and set in motion a process to reattach itself to the newly created browser. This process is far from perfect. It is subject to really challenging race conditions that are truly impossible to eliminate entirely, because of the architecture of the browser itself. Nevertheless, even in its flawed state, this is still a better outcome than it was previously for users.<br /><br />Please note that the advice and support policy of the IE driver will continue to be that the user must set the Protected Mode settings of the browser properly before using the driver. Any "issues" that arise by not having the settings set, but that disappear when the settings<br />are corrected, are not considered by the project to be valid issues. This will include, but not be limited to, issues like abandoned browser instances not being closed, use of multiple instances of the driver where the wrong browser window is connected to and automated, and issues where the driver appears to hang upon navigation to a new page. If the problem disappears when the browser is properly configured, any issue reports will be immediately closed with a note to properly configure the browser and remove the capability.<br /><br />The following situations should be at least partially mitigated by the change:</span><br />
<ul style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;"> Navigation to a new page</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;"> Clicking on a link (specifically an <a> tag) that will lead to navigation to a new page</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;"> Clicking a link that opens a new window</span><span style="font-family: Arial, Helvetica, sans-serif;"></span><br /><span style="font-family: Arial, Helvetica, sans-serif;"></span></li>
</ul>
<span style="font-family: Arial, Helvetica, sans-serif;">Other cases, like navigating backward and forward through the browser history, clicking an element that submits a form, and so on, may not be handled. In those cases, issue reports will be summarily closed, unless a specific pull request fixing the issue is also provided. Additionally, use of things like proxies to capture traffic between the browser and web server may miss some traffic because of the race conditions inherent in the mechanism used to reattach to a newly created browser. Again, these race conditions are unavoidable, and<br />issue reports that are based on them will be immediately closed with a note indicating that the browser must have its settings properly set. These strict guidelines are not intended to be harsh, and are not put in place with the intent to avoid investigating and fixing issues;<br />rather, they must be enforced because the underlying architecture of the browser makes them unavoidable.<br /><br />While not perfect, it's hoped that these changes will make things a little easier for users who run against Internet Explorer, but are prevented by circumstances beyond their control from properly configuring the browser. If you're one of those unlucky users, I hope you'll give the driver a spin, and see how it works for you.</span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com0tag:blogger.com,1999:blog-7433739558862985636.post-81740211811001634392018-03-11T19:36:00.003-07:002018-03-11T19:37:43.718-07:00Deprecating Parts of Selenium's .NET Bindings<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Note:</b> This blog post should be considered a work-in-progress until this note is removed. </span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">With the release of 3.11.0 of the Selenium .NET bindings, a few things in the support library (WebDriver.Support.dll) have been marked with the Obsolete attribute. This will come as something of a surprise for some users when they update to that version. In particular, the .NET implementation of the PageFactory and the ExpectedConditions class used with WebDriverWait have been deprecated. It's understandable that there would be some consternation about suddenly seeing compile warnings mentioning removing components in a future release of Selenium, particularly if one's own code makes use of those components. Why would the .NET bindings' maintainers do this?</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">First, one must consider the original intent of the WebDriver.Support.dll assembly. When originally created, it was originally designed to showcase some of the things that would be possible to create based on the WebDriver API. It was not intended that large numbers of users would plug those examples directly into production code.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Second, the .NET implementation of these constructs was created mostly because some users asked, "Java has it, so why doesn't .NET?" Rather than blindly copying the Java implementations as was done, it would have been better to think about what actually makes sense when using C#. In other words, "C# isn't Java, and therefore the things that work best for Java may not be entirely appropriate for C#."</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">In the case of the .NET PageFactory, the implementation was problematic and cumbersome, as well as not nearly flexible enough for the myriad ways people wanted to create Page Objects. Additionally, when .NET Core 2.0 was released, the classes upon which the .NET PageFactory relied were not included .NET Core 2.0. This meant that to get the PageFactory working under .NET Core, the project either had to take on a new dependency, mangle the code with conditional compile directives, or leave it unsupported in .NET Core. The first approach is a non-starter for the Selenium project's .NET bindings, the reasons for which should be a subject of its own blog post. The second approach made the code nearly impossible to properly maintain.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Furthermore, with respect to the PageFactory in particular, there is no benefit to be gained by identifying elements via an attribute over doing it directly in runtime code. Claims that the PageFactory made Page Object creation and maintenance less verbose simply do not hold up under close scrutiny.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">With respect to ExpectedConditions, again, this was an addition that was created in .NET solely because "Java has it." At the time the ExpectedConditions class in Java was created, the syntax for creating a lambda function (or something that acted like one) was particularly arcane and difficult to understand. In that case, a helper class made lots of sense for the Java bindings. </span><span style="font-family: "arial" , "helvetica" , sans-serif;"> However, C# isn't Java. In C#, the syntax for creating lambda functions ("anonymous methods" in the language of Microsoft's documentation) has been well understood by C# developers for many years, and is a standard tool in their arsenal.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In this case, the question of code verbosity does have some merit, but since wait conditions are rarely one-size-fits-all, it would be a much cleaner approach for users to develop their own conditions class that has the wait conditions they're interested in. This, however, is something users have an aversion to. Additionally, the thought of a "standard" collection of implementations of specific wait conditions seems to be a good idea on its face, but there is a great deal of variation on the way users want any given condition to work. Having a collection of wait conditions might be a good thing, but the Selenium project is not the place for it.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">So that people would still have access to the existing implementations, a new organization has been set up on Github. The code for these two pieces of the support library have been migrated there, and the first binary artifacts have been distributed concurrent with Selenium 3.11. People can move over to the ported implementations with minimal change (usually just a namespace change) to their own code. The new repo is awaiting someone from the community who feels strongly about maintaining these types of libraries. </span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com8tag:blogger.com,1999:blog-7433739558862985636.post-26438181652772613222017-09-25T10:10:00.003-07:002017-09-25T10:34:26.944-07:00Selenium WebDriver Support For .NET Core 2.0<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Starting with release 3.6.0 of the .NET bindings, Selenium now has the initial support for .NET Core 2.0. The .NET bindings in that release contain versions of the assemblies that are build against the .NET Standard 2.0 platform, which means they're intended to be used with .NET Core 2.0 projects. I know this has been a feature many people have wanted for a long time, and I'm glad the project can now deliver it. However, it does come with some associated costs, and with a few known issues.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The first known issue is that calls to localhost in .NET Core are slower than those in the full .NET Framework.</span> <span style="font-family: "arial" , "helvetica" , sans-serif;">This is due to internal differences in the .NET libraries</span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> themselves, and are not the fault of the bindings directly. See</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> <a href="https://github.com/dotnet/corefx/issues/24104">this issue in the .NET Core repository</a> for more details</span></span><br />
<br />
<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Secondly, attempting</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> to save a screenshot to any graphics file format other than</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> Portable Network Graphics (PNG) will throw an exception. .NET Core does </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">not provide the image manipulation classes that the full .NET Framework </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">does, and there are no production-ready third-party libraries that </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">provide that functionality yet and also only rely on managed code. It's fully possible to save a screenshot when using .NET Core, but you can only save it to the PNG file format within the Selenium libraries. This </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">concern is over and above the difficulties with adding dependencies to </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">the language bindings</span></span><br />
<br />
<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Speaking of difficulties with adding dependencies to the Selenium project leads me to the next known issue. </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">When using the bindings against .NET Core, there is no PageFactory </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">class available. This is not an oversight, nor is it a bug. I have long said that the .NET PageFactory implementation is not required for effective implementation of the Page Object Pattern, and the .NET PageFactory does not provide any tangible benefits to the user. Even the argument that the code is easier to read is specious with properly constructed page objects. Moreover, the existing .NET </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">PageFactory implementation requires use of classes that are not </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">available in .NET Core. It is a non-trivial matter to add additional </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">dependencies to the .NET bindings, so simply replacing those classes </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">with a third-party library that is compatible with .NET Core is not a </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">"perfectly obvious" option.</span></span><br />
<br />
<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Finally, r</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">eferences to the .NET Standard 2.0 library versions provided in this</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> and future releases are only valid when using NuGet package references.</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> Simply copying the assembly and adding an assembly reference to the</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> .NET Core 2.0 project will not work. This is by design of the .NET</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> Core ecosystem, which is now entirely dependent on NuGet to propertly</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> resolve dependencies.</span></span><br />
<br />
<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">One last note with the 3.6.0 release of the .NET bindings. Previously, the .zip archives that were provided at the <a href="http://selenium-release.storage.googleapis.com/index.html">official Selenium release site</a> contained only the assemblies (.dlls) for the various frameworks that we supported. Starting with this release, the downloadable .zip archives contain NuGet package (.nupkg) files inside the .zip. To extract the actual .dlls from the packages, you can use any .zip reader to extract files from a .nupkg file. Yes, this means that we're putting a .zip inside a .zip, which is less than efficient, and we may revisit this mechanism of distributing the binary releases in the future.</span></span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.comtag:blogger.com,1999:blog-7433739558862985636.post-21255754009665921222017-03-22T13:03:00.000-07:002017-03-22T13:38:56.770-07:00Announcing Beta Release of Selenium IE Driver<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">One of the most common question I get asked is, "How can I help contribute to Selenium?" Usually my answer involves pull requests and the like, but today, I can offer a much easier way for people to contribute. A significant part of my attention over the last four years has been thinking about and working on the <a href="https://w3c.github.io/webdriver/webdriver-spec.html">W3C specification for WebDriver</a>. While the specification codifies many of the things that the open source Selenium project has done for years, there are some significant changes to the wire protocol that the language bindings use to communicate with the drivers themselves. The specification already has an implementation in wide use, in <a href="https://github.com/mozilla/geckodriver">geckodriver</a>, Mozilla's driver implementation for Firefox. In order to move forward, however, the IE driver needs to be updated to follow the specification. Here's where you come in.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">I've modified the IE driver to use the W3C dialect of the wire protocol. This modification, while significant internally, shouldn't show any differences in behavior from the existing, shipping IE driver. It currently passes all of the tests in the Selenium project for IE. While these tests are pretty extensive, the permutations available in the DOM and in Selenium WebDriver code used to automate it are nearly infinite. To that end, I'm announcing the availability of a beta version of the IE driver. What am I asking you to do? Simply <a href="http://selenium-release.storage.googleapis.com/index.html?path=IE.Driver.Beta/">download the new driver executable</a>, and use it in place of the existing driver you're using in your Internet Explorer automation.</span><br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Notes</span></h3>
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">The beta driver should be a drop-in replacement for the existing 3.3.0 IEDriverServer.exe release. It should require no changes in your code, save maybe pointing to the new executable.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Having said that, there <span style="font-family: "arial" , "helvetica" , sans-serif;">are some differences that are expected due to spec compl<span style="font-family: "arial" , "helvetica" , sans-serif;">iance. Full-page screenshots<span style="font-family: "arial" , "helvetica" , sans-serif;">, for example, are explicitly disallowed by the <span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">specification</span>, so are no longer generated by the driver.</span></span></span></span> </span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">The beta driver's version number (visible by executing <span style="font-family: "courier new" , "courier" , monospace;">IEDriverServer.exe --version</span>) will be 3.3.99.x. Bug fix releases will increment the "build" (fourth) field of the version number.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">This executable will only be available via <a href="http://selenium-release.storage.googleapis.com/index.html">the download site</a>; it will not be available via package managers (<a href="https://mvnrepository.com/">Maven</a>, <a href="https://www.nuget.org/">NuGet</a>, <a href="https://www.npmjs.com/">npm</a>, etc.). If the beta appears in any of the (unofficial) packages that may be used for IEDriverServer.exe in a package manager, a request will be sent to the package owner to remove it, so please don't rely on those.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">There have been some extensive internal rewrites due to the nature of the protocol changes. More on what to look for below.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Only the 32-bit version of the driver is being provided for the beta. </span></li>
</ul>
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Areas of Concern</span></h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We want to know if there are any differences between the shipping 3.3.0 version of IEDriverServer.exe and the beta version. You should see the same behavior, including bugs; do not expect the beta driver to magically fix issues you may have experienced with IE in the past. Updating to support the specification wire protocol has required extensive rewrites, but these should all be transparent to the language bindings. The biggest changes have happened in the areas of element interactions, so you should pay special attention to things like <span style="font-family: "courier new" , "courier" , monospace;">WebElement.click()</span> or <span style="font-family: "courier new" , "courier" , monospace;">WebElement.sendKeys()</span>. There is one known issue that if you call <span style="font-family: "courier new" , "courier" , monospace;">WebElement.submit()</span>, and the onsubmit event handler throws up a JavaScript alert(), the driver will hang. This issue won't be fixed until after the merge back to master.<span style="font-family: "arial" , "helvetica" , sans-serif;"> </span>Also note that the beta has to date only been tested<span style="font-family: "arial" , "helvetica" , sans-serif;"> </span>against IE 11, and per the driver's<span style="font-family: "arial" , "helvetica" , sans-serif;"> </span>current support policy, only officially<span style="font-family: "arial" , "helvetica" , sans-serif;"> </span>supports IE 9, 10, and 11.</span><br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Reporting Issues</span></h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Issues with the beta can be reported to the <a href="https://github.com/SeleniumHQ/selenium/issues/new">Selenium project's issue tracker</a>. However, we have to set some ground rules for the issues that you submit. Here they are:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">You'll need to provide the following information with any issue report:</span><br />
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Language bindings (Java, .NET, Ruby, Python, JavaScript) and version number you're using</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">The specific version of the beta you're using</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">The WebDriver code that behaves differently</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">An HTML page (or link to one) that the WebDriver code can be run against </span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Lack of any of this information will cause the issue to be closed immediately, without action or investigation!</b> There are simply too many other potential issues with the existing IE driver, and the timeline for getting this merged into the main code line is simply too short to be able to go back and forth with issue reporters trying to set up a reproducible case. Moreover, here are some further guidelines about submitting issues.</span><br />
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Prefixing your issue title with "IE Driver Beta" will get it processed more quickly than if you don't. </span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">The beta has only been tested with 3.3.x versions of any language bindings. It should still work with any language bindings of the 3.x vintage, but if you haven't tried your code with at least 3.3.x, you will be asked to do so before further investigation can continue on your issue.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">You should be able to concretely demonstrate a difference in behavior from IEDriverServer.exe 3.3.0 and the beta you're using. If you cannot, you will be asked to do so before investigation can continue.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">If you are using a test framework, and your sample code cannot be extracted to simple, straightforward WebDriver-only code, your issue will be closed. Developer bandwidth is just too narrow to wade through tons of framework code to get to the single few lines of WebDriver code that are exhibiting different behavior.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">If you omit an HTML page that can be tested against, your issue will be closed. Again, this may seem overly restrictive, but without this caveat, it will be nearly impossible to debug the issue with the beta driver.</span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;">This is pretty time-sensitive, so if you'd like to give this a try, the Selenium project developers would really appreciate it. </span><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
</div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com2tag:blogger.com,1999:blog-7433739558862985636.post-14225847581538276862017-02-13T09:18:00.005-08:002017-02-13T09:23:28.958-08:00Announcing End of Life of .NET Selenium RC Language Bindings<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;">This post will serve as the official announcement that version 3.1 of the Selenium .NET language bindings will be the last to provide a Selenium RC library. Users still relying on the RC API will be able to continue to do so using WebDriverBackedSelenium, or by converting your code to use WebDriver proper. Selenium RC has been deprecated for over six years, and the .NET Selenium RC language bindings have not been updated with a code change other than a version bump in nearly that long. This change isn't likely to affect many users at this point, and the 3.1 versions of the language bindings will continue to be available more-or-less indefinitely, but there will be no further changes to the .NET RC library or releases of it.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Let me <span style="font-family: "arial" , "helvetica" , sans-serif;">restate a<span style="font-family: "arial" , "helvetica" , sans-serif;">gain so th<span style="font-family: "arial" , "helvetica" , sans-serif;">at it's blata<span style="font-family: "arial" , "helvetica" , sans-serif;">ntly obvious. This d<span style="font-family: "arial" , "helvetica" , sans-serif;">oes not affect the <span style="font-family: "arial" , "helvetica" , sans-serif;">.NET language bindings for WebDriver, and WebDriverBackedSelenium will remain a viable path forward for some time. It only affects <span style="font-family: "arial" , "helvetica" , sans-serif;">Selenium RC<span style="font-family: "arial" , "helvetica" , sans-serif;"> in th<span style="font-family: "arial" , "helvetica" , sans-serif;">e .NET <span style="font-family: "arial" , "helvetica" , sans-serif;">language bindings.</span></span></span></span></span></span></span></span></span></span> </span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com1tag:blogger.com,1999:blog-7433739558862985636.post-3518084843411668202016-08-23T10:37:00.000-07:002016-08-23T10:37:50.023-07:00Polyamory, Pride Flags, and Patterns of Feedback<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Warning:</b> For those of you who come here looking for technical advice and inside information about the Selenium project, WebDriver, or browser automation, this post isn't about any of those. You might just want to skip this one altogether.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">One thing about me I'm not really sure how many people are aware of is that I'm <a href="https://en.wikipedia.org/wiki/Polyamory">polyamorous</a>. That means that I am comfortabl<span style="font-family: "arial" , "helvetica" , sans-serif;">e</span> being in simultaneous romantic relationships with multiple partners at once, and that my participation in those relationships is openly known by all people involved. I've been polyamorous, or "poly" for short, for nearly all of my adult life. A little over 20 years ago, I lived in the Pacific Northwest, and for the first time in my life, I experienced first-hand the struggles and celebrations of what is now known as the LGBT community. One thing that struck me was the imagery and symbolism those communities used to rally around, identify other members, and publicly announce their membership in the community. The pride flag was one image that made a huge impression on me. At that time, the poly community didn't really have similar symbols to use, so I took it upon myself to create one. Here's what I made up, and released into the public domain in the late summer or early fall of 1995.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div style="text-align: center;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSkYPSy_T4W0i5QiZ2pfRyvijTQNqDhdVr76151BhgBqTDB3OBx4sNL2EjY_cVIGnJFfyI3Ro3Pbe4NoPLtvtmpRQKmiz_6EW54mm-S6HPpVxWZ6nVSMY7Cyh8cpnODaOehU0bLf0CWyU/s1600/polyamory-flag.png" imageanchor="1"><img border="0" height="212" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSkYPSy_T4W0i5QiZ2pfRyvijTQNqDhdVr76151BhgBqTDB3OBx4sNL2EjY_cVIGnJFfyI3Ro3Pbe4NoPLtvtmpRQKmiz_6EW54mm-S6HPpVxWZ6nVSMY7Cyh8cpnODaOehU0bLf0CWyU/s320/polyamory-flag.png" width="320" /></a></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Here's the text I wrote up describing it to the first mailing list I shared it with. It's become the canonical description of this particular flag.</span><br />
<blockquote class="tr_bq">
<span style="font-family: "arial" , "helvetica" , sans-serif;">The poly pride flag consists of three equal horizontal colored stripes
with a symbol in the center of the flag. The colors of the stripes, from
top to bottom, are as follows: blue, representing the openness and
honesty among all partners with which we conduct our multiple
relationships; red, representing love and passion; and black,
representing solidarity with those who, though they are open and honest
with all participants of their relationships, must hide those
relationships from the outside world due to societal pressures. The
symbol in the center of the flag is a gold Greek lowercase letter 'pi',
as the first letter of 'polyamory'. The letter's gold color represents
the value that we place on the emotional attachment to others, be the
relationship friendly or romantic in nature, as opposed to merely
primarily physical relationships.</span></blockquote>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Now, here are some things to understand. Clearly, I'm not a visual artist. My tools for creation at the time were literally limited to Microsoft Paint, running on Windows 3.1. Nevertheless, the flag design managed to limp along, with little fanfare. My friends and I used it, and thought of it as quirky and something that could be used in the way other pride flags were used, as a symbol to rally around and for identification.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Fast forward 20 years. Apparently, this thing called the World Wide Web happened, and let all sorts of people communicate and discover things they'd never known about before. New polyamorous people began to discover the flag existed. One would think that people might think it was an interesting idea, given its intent. One would be wrong. The flag has been called <a href="https://www.reddit.com/r/polyamory/comments/4e5wfn/controversial_opinion_time_i_hate_the_flag/d1xq48h">vile</a>, <a href="https://www.reddit.com/r/polyamory/comments/4e5wfn/controversial_opinion_time_i_hate_the_flag/d1yck4t">no good</a>, <a href="https://www.reddit.com/r/polyamory/comments/4e5wfn/controversial_opinion_time_i_hate_the_flag/d1xgie6">hideous</a>, <a href="http://pride-flags-for-us.tumblr.com/post/92066912269/ive-recently-come-to-terms-with-the-fact-that-i">disappointing</a>, <a href="https://www.reddit.com/r/polyamory/comments/v9d96/so_apparently_this_is_the_polyamory_pride_flag/?ref=search_posts">ugly</a>, and many other negative things.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">One of the issues frequently brought up is that the color scheme is garish or unpleasing. That's subjective, and I can't argue with their perception. I still think there's value in the color
symbology, if not the actual RGB values I used when creating it.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Many people seem to take issue with the pi symbol as obscure. There
were specific reasons for choosing it at the time. First, I specifically
avoided imagery that included a heart. The <a href="https://en.wikipedia.org/wiki/Leather_Pride_flag">leather pride flag</a>, which
predates the design of mine, includes a heart, and I was trying to avoid
confusion, given that community was there first. The <a href="https://en.wikipedia.org/wiki/Polyamory#Symbols">"infinity heart"</a>
was not yet as widely accepted a symbol for polyamory, and would have
been challenging for me to incorporate given my limited abilities in the
visual arts. The letter pi was readily available on computer
typographic platforms even in those days, so I chose that.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Also, at the time, I was more concerned with "in the closet"
polyfolk, and was far more in the closet myself than I am these days. I
wanted a symbol that could be used relatively anonymously, that could
let people who were in on the symbology connect, without it being too
specific.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Additionally, there was already a rich history of existing pride
symbols using Greek letters, the use of <a href="https://en.wikipedia.org/wiki/LGBT_symbols#Lambda">lambda as an LGBT symbol</a>, being a concrete example. I was
hoping to evoke similarity and solidarity without being too explicit or
derivative. Finally, the fact that the "poly" in polyamory is a Greek
root seemed to indicate that would be a natural choice. In retrospect,
perhaps a <a href="https://en.wikipedia.org/wiki/Lemniscate">lemniscate</a> ("infinity symbol") would've been a better choice,
but nobody spoke up then.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Poly people coming to read this full story for the first time, welcome. Glad to meet you. If you don't care for the flag, I'm sorry to have offended your sensibilities. Today, there are a number of alternative symbols you can rally around. Use mine, don't use it, I'm
just glad some people found a banner to rally around in the late '90s. Feel free to leave comments, but dismissive and abusive comments will be removed.</span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com6tag:blogger.com,1999:blog-7433739558862985636.post-61466330308770146182015-07-23T12:27:00.003-07:002015-07-23T13:18:56.278-07:00Using the Microsoft Driver for Microsoft Edge<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibpMYa5nM6WXquUyBVvosBj2Qvm288Tc_Bc9ipzfg2PKaG3Jvyt4BgWTuWe_FV8ax2lJztGFqgPV6qHqzU_BoVhT-RyKlqvYvjZrlTmSbTWOrWd2K3l0C9UpEEfxJ3_WXJlfNowm5UIAE/s1600/830704.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibpMYa5nM6WXquUyBVvosBj2Qvm288Tc_Bc9ipzfg2PKaG3Jvyt4BgWTuWe_FV8ax2lJztGFqgPV6qHqzU_BoVhT-RyKlqvYvjZrlTmSbTWOrWd2K3l0C9UpEEfxJ3_WXJlfNowm5UIAE/s320/830704.jpg" width="320" /></a></div>
<br />
<br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">As the release of Windows 10 has approached, I've been seeing <a href="https://groups.google.com/forum/#!topic/selenium-users/A-CxYej5FLk">more</a> and <a href="http://stackoverflow.com/questions/30044912/is-there-a-selenium-webdriver-available-for-the-microsoft-edge-browser">more</a> questions about a WebDriver implementation for Microsoft Edge (nee "Spartan"). So far, I've only been able to say that there isn't a driver implementation for that particular browser, and that there is no work being done in the open-source arena on such an implementation. Furthermore, that Microsoft has acknowledged the need for such a driver and has <a href="http://dev.modern.ie/platform/faq/webdriver">committed to providing one</a>, but has provided no timetable for a release of such an implementation.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span>
<span style="font-family: Arial,Helvetica,sans-serif;">As of today, I can say much more than that.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Today, Microsoft has <a href="http://blogs.windows.com/msedgedev/2015/07/23/bringing-automated-testing-to-microsoft-edge-through-webdriver/">announced the availability</a> of a WebDriver implementation for Microsoft Edge. This implementation is released as an installable application for Windows 10, and you can download the installation package <a href="http://www.microsoft.com/en-us/download/details.aspx?id=48212">here</a>. While the requirement to run an installer might be off-putting to some users, it appears that the installation package merely installs a standalone executable to the restricted "Program Files" location. As near as I can tell by examining the installer package, the executable can be freely copied to other locations on the machine after installation. Also, the use of an installable package means the executable can be serviced and updated by the standard, automatic Microsoft Update mechanism.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">The last point is incredibly important. This first release of a driver from Microsoft is most emphatically not a finished release, and as such it lacks functionality, some of it rather basic. To wit, finding an element within the context of another element (i.e., <span style="font-family: "Courier New",Courier,monospace;">WebElement.findElement</span>) is not yet implemented in the current release. Finding an element via XPath is not implemented in this first release either. Switching to frames or iframes (i.e., <span style="font-family: "Courier New",Courier,monospace;">driver.switchTo().frame()</span>) is likewise not implemented in this initial release, and neither is the advanced user interactions API (i.e,. the <span style="font-family: "Courier New",Courier,monospace;">Actions</span> class). In fairness, <a href="http://dev.modern.ie/platform/status/webdriver/details/">Microsoft has been completely forthcoming</a> in what functionality the driver has implemented and what is missing. I fully expect that the driver will be receiving regular updates, and that the missing functionality will be added in the coming weeks and months.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">So, how does one use the Microsoft Edge driver? Well, thanks to our good friends in Redmond, the release of the driver implementation was accompanied by a <a href="https://github.com/SeleniumHQ/selenium/pull/820">pull request to the Selenium project</a> that enables the existing language bindings to use it seamlessly. The Edge driver as currently released uses the open-source project's dialect of the <a href="https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol">WebDriver JSON wire protocol</a>, allowing us to use the driver right now. My guess would be that a future release of the driver will use the <a href="https://w3c.github.io/webdriver/webdriver-spec.html">W3C specification</a> version of the protocol. The pull request has been merged, so all you'll need is an updated language binding release, and you'll be able to use it directly. If you're itching to use it in the meantime, you can manually launch <span style="font-family: "Courier New",Courier,monospace;">MicrosoftWebDriver.exe</span>, and use the <span style="font-family: "Courier New",Courier,monospace;">RemoteWebDriver</span> class in your current language bindings to connect to it on your local Windows 10 machine.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Congratulations to the Microsoft Edge development team for releasing a standard tool for automating Microsoft Edge. I, for one, appreciate the hard work and efforts you've put forth to make this happen. I look forward to future enhancements and features, and I stand ready to help any way I possibly can.</span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com3tag:blogger.com,1999:blog-7433739558862985636.post-82846673042303104962014-12-22T10:45:00.001-08:002015-02-17T09:06:36.613-08:00Windows Update KB3025390 for IE 11 Breaks IE Driver<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial,Helvetica,sans-serif;"><b>Update (10 February 2015):</b> Microsoft has released a fix as part of the <a href="https://support.microsoft.com/kb/3021952">February 2015 Cumulative Update to Internet Explorer</a>. Installing this update appears to resolve the issue with the IE driver.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">On 16 December 2014, Microsoft released update <a href="https://support.microsoft.com/kb/3025390">KB3025390</a> via Windows
Update as part of its normal "patch Tuesday" update cycle. For most
users, this update is downloaded and installed without user interaction.
This update breaks the IE driver when using it with IE11.
<br />
<br />As part of this update, attempting to use the COM method
<a href="http://msdn.microsoft.com/en-us/library/ie/ms536420%28v=vs.85%29.aspx">IHTMLWindow2::execScript</a> returns an "access denied" result. This
renders the driver unable to execute JavaScript in the page bring
browsed. However, given that large portions of driver functionality are
implemented using JavaScript, this effectively renders the driver all
but unusable with IE11.
<br />
<br />There is no known workaround for this issue. At this time, <a href="http://jimevansmusic.blogspot.com/2014/09/using-internet-explorer-webdriver.html">Microsoft's WebDriver implementation</a> for IE is still incomplete, lacking basic
functionality required to make it usable, so it cannot be recommended.
Uninstalling the update is reported to restore IE driver functionality, but this
is hardly ideal.
<br />
<br />While the execScript method is <a href="http://msdn.microsoft.com/en-us/library/ie/bg182625%28v=vs.85%29.aspx#legacyAPIs">marked as deprecated</a> for IE11, the
driver had heretofore been able to use that method successfully, and it
was hoped that it would remain useful throughout the IE11 life cycle. We
now know this not to be the case. Additionally, attempts to use the
Microsoft-suggested replacement, <span style="font-family: "Courier New",Courier,monospace;">eval</span>, have been fruitless thus far.
<br />
<br />At the moment, a bug has been raised via <a href="https://connect.microsoft.com/IE/feedback/details/1062093/installation-of-kb3025390-breaks-out-of-process-javascript-execution-in-ie11">Microsoft Connect</a> on this issue, and is being investigated by the Internet Explorer development team. </span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-family: Arial,Helvetica,sans-serif;">The issue is also currently being tracked in the <a href="https://code.google.com/p/selenium/issues/detail?id=8302">Selenium issue tracker</a>. There is no need to post additional comments in that issue
report verifying that you are experiencing the issue. Likewise, there is
no need to post comments to the issue asking for status updates. </span>Rest
assured that the issue in the Selenium tracker will be updated when new information is
discovered.
<br />
<br />What can you do to help? In the coming days and weeks, I'm sure we will see a
large number of people exclaiming that their WebDriver code has
mysteriously stopped working against IE11, with no action on their part,
and will not have searched for answers. I'm posting this message
several places as a public service announcement, and would like all of
you to redirect such inquiries to this post.
</span><br />
<a href="https://code.google.com/p/selenium/issues/detail?id=8302" target="_blank"></a></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com4tag:blogger.com,1999:blog-7433739558862985636.post-6060257580816531462014-09-16T16:41:00.000-07:002016-06-23T10:57:46.871-07:00Screenshots, SendKeys, and Sixty-Four Bits<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCAAqUSUotNOtk60waHSMWjG-wckHeOjqcaue75uoeEikJMfBXlITJSBU3SIZOUCKDj-wPUNqU_hyphenhyphenhFzawBdxOeiWozFKpebGxQymMygyfbmXCyfRaJd6vgn8O-VIEkFNwqKyztzIoMXI/s1600/p0PqoeY.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCAAqUSUotNOtk60waHSMWjG-wckHeOjqcaue75uoeEikJMfBXlITJSBU3SIZOUCKDj-wPUNqU_hyphenhyphenhFzawBdxOeiWozFKpebGxQymMygyfbmXCyfRaJd6vgn8O-VIEkFNwqKyztzIoMXI/s1600/p0PqoeY.jpg" width="320" /></a></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">There are a couple of issues with the Internet Explorer driver that have been around since IE10 was released. They're pretty annoying when people encounter them, and the report for the first issue goes something like this:</span><br />
<blockquote class="tr_bq">
<span style="font-family: "arial" , "helvetica" , sans-serif;">I'm using Internet Explorer 10 (or 11), and when I call the sendKeys method, the keystrokes happen very slowly. Like one keystroke every 5 seconds. I'm on a 64-bit version of Windows, and I'm using the 64-bit IEDriverServer.exe. If I use the 32-bit version of the driver, the problem doesn't occur, but I really need to test using the 64-bit version because I need to test 64-bit IE. What's the deal?</span></blockquote>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The report for the second issue usually reads as follows:</span><br />
<blockquote class="tr_bq">
<span style="font-family: "arial" , "helvetica" , sans-serif;">I'm using Internet Explorer 10 (or 11), and even though I'm on 64-bit Windows, I'm using the 32-bit IEDriverServer.exe, because I was having problems with sendKeys being slow. Now, though, when I take a screen shot, it only shows the visible portion of the page. How can I take full page screen shots like I could when I use the 64-bit IEDriverServer.exe?</span></blockquote>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Both of these issues are fully documented in the Selenium issue tracker (<a href="https://code.google.com/p/selenium/issues/detail?id=5116">#5116</a> for the sendKeys issue, and <a href="https://code.google.com/p/selenium/issues/detail?id=5876">#5876</a> for the screenshot issue). A comment in each issue mentions that any fix would require "a massive rearchitecture of the IE driver's binary components, [so] no timeline is (or will be) available" for the delivery of a fix. What causes these issues? How are they related? Why would a fix be so darned difficult? The answers to those questions can all be summed up with a simple answer: "Windows Hooks." </span><br />
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">What is a Windows Hook?</span></h3>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">All Windows applications have a routine in them called a "message loop." The message loop repeatedly calls the <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms644936%28v=vs.85%29.aspx">GetMessage</a> API function, and processes messages sent to the application as they arrive in its queue. Hooks are a feature of the Windows message handling system that allow a developer to intercept, examine, and modify the message being sent to the application. By installing a hook, a developer could, for example, validate that a certain message was processed by the window being hooked. Or they could modify a message sent to the window to represent that the operating system could do things it actually can't. It's a clever mechanism, but it does have a few requirements.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">First of all, the code being run when the hook is called (the "hook procedure") must exist in a dynamic-link library (DLL). That is, it cannot be simply a function exported from a compiled executable. The reason for this is that the code is actually going to be loaded into two applications, the application installing the hook, and the application being hooked. Using a DLL is the only way to avoid certain conflicts that would arise with loading one executable into the process space of another.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Secondly, the DLL must be of the same "bitness" of the process being hooked. In Windows, a 32-bit executable cannot load a 64-bit DLL. The converse is also true, that a 64-bit executable cannot load a 32-bit DLL. Incidentally, this is the root reason that there are two versions of IEDriverServer.exe, but that's another story for another time.</span><br />
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Windows Hooks and the IE Driver</span></h3>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The IEDriverServer.exe uses hooks for its implementation of a couple of features. The first use of a hook is in the processing of keystrokes. By default, the driver uses the Windows <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms644944%28v=vs.85%29.aspx">PostMessage</a> API function to simulate keystrokes. It does this by sending a WM_KEYDOWN message, followed by WM_CHAR and WM_KEYUP messages for each key. However, PostMessage is asynchronous, so the driver has to wait to make sure that the WM_KEYDOWN message is processed before sending the other messages, otherwise keystrokes could be sent out of order, making key sequences garbled. The driver does this by installing a hook into IE's window procedure, and listening for the WM_KEYDOWN to be processed before proceeding. It also puts in a timeout of about five seconds waiting for the message to be processed to make sure that the driver doesn't wait forever. Note that the code path is slightly different if you're using the requireWindowFocus capability, using the <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms646310%28v=vs.85%29.aspx">SendInput</a> API function instead, but the driver still uses a hook to make sure messages are processed before moving on.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The second place the driver uses a hook is when taking screenshots. The IE driver takes screenshots using the <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/dd162869%28v=vs.85%29.aspx">PrintWindow</a> API function. PrintWindow can only take a screenshot of the visible portion of any given window, which means that in order to get a full-page screenshot (as required by the WebDriver API), the window must be sized large enough to display the entire page without scroll bars. However, Windows does not allow the window to be resized larger than the visible screen resolution. When we ask IE to resize itself, a WM_GETMINMAXINFO message is sent on a resize event so the IE can figure how large a window can be. By intercepting that message with a hook, and modifying the max values, we can trick IE into thinking that a window can be sized greater than the screen resolution would otherwise allow.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Since the IE driver makes use of hook procedures, the bulk of the IE driver is actually implemented in a DLL. So as to avoid having to manage multiple files when using the IE driver, this DLL is embedded as a resource inside the IEDriverServer executable, and extracted to the temp directory at runtime. Once extracted, it's loaded into memory by IEDriverServer, and the main entry point of the DLL is called. This gives the driver a way to inject itself into the IE process using hooks to accomplish what it needs to. This worked great up to and including IE 9.</span><br />
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">What Happened in IE10?</span></h3>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">When IE 10 was first released, we started to see reports of the two aforementioned issues coming in. Since version 7 of Internet Explorer, there has been the notion of multiple processes for a single "instance" of IE. There was the notion of a "manager" or "broker" process, which managed the outer, top-level window of Internet Explorer. All HTML rendering and ActiveX controls are managed by a "content" process. Through version 9 of Internet Explorer, these processes were the same bitness. That is, running 64-bit IE meant you got a 64-bit manager process, and 64-bit content processes. Running 32-bit IE meant you were using a 32-bit manager process, and 32-bit content processes. This all changed with IE 10.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">One major change in IE 10 is that the manager process on 64-bit versions of Windows will always be a 64-bit process. By default, though, content processes remained 32-bit. This allowed the main process to be a 64-bit process, but still allowed the browser to remain compatible with all of the existing browser plug-ins for IE, which are overwhelmingly 32-bit. There are other reasons for this change, and I'm oversimplifying the architecture a bit. If you're truly interested in the deep details of the architecture, I encourage you to read <a href="https://blogs.msdn.microsoft.com/ieinternals/2012/03/23/understanding-enhanced-protected-mode/">Eric Lawrence's blog post</a> that lays out the implications in great detail. Anyway, there are ways to force 64-bit content processes for IE 10 and above, but these will break the IE driver due to Protected Mode issues.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">So for IE 10 and above, the situation is that we have a 64-bit process, which handles the main outer window, and a 32-bit process which owns the inner window where HTML content is rendered. The driver's window hook procedure for taking screenshots must be attached to the main outer window; the driver's hook procedure for verification of message processing while simulating keystrokes must be attached to the content window being automated. Remember that since the driver executable can only be 32-bit or 64-bit, but not both, the DLL in which the window hook procedures reside can only be the same bitness as the executable. Let's explore the implications of this.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">For the 32-bit IEDriverServer, the hook procedure can be successfully attached to the (32-bit) content window for use with sendKeys. Attempting to install the window hook for screenshots is attempting to install a window hook into the manager process, which is 64-bit, which can't load the 32-bit DLL into its process space, so the hook installation fails, and screenshots are truncated.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Conversely, for the 64-bit IEDriverServer, the hook can be successfully installed in the top-level window for use in taking screenshots, because the process owning that window is a 64-bit process. However, when the driver attempts to install the hook into the (32-bit) content window to detect message processing during sendKeys, the DLL is 64-bit, and can't be loaded by the 32-bit executable which owns the content window. This means that the timeout is invoked for every keystroke, with sendKeys waiting about five seconds for each key.</span><br />
<br />
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Why Would This Be So Hard To Fix?</span></h3>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">By now, I hope we have a better understanding of what the root cause of both issues is. What would it take to fix the issue? A naive implementation would just attempt to bundle both 32- and 64-bit DLLs in with a version of the server and be done with it. However, this won't work because the DLL where the hook procedure lives must be loaded by <i>both</i> executables, IEDriverServer.exe, and the IE process owning whichever window is being hooked. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The only way to completely and correctly resolve the issue would be to create a pair of executables, with a related pair of DLLs, and have the two executables establish some way of working together via an interprocess communication channel. With this approach, now we'd be asking a user to download and manage two executables instead of one, or to use an installer of some kind. Since the project aims for an "xcopy deploy" without requiring the use of an installer, </span><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">that's a larger burden than one can expect users of the IE driver to undertake.</span></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">While it's tempting to simply suggest creating a second executable and embedding it as a resource for extraction at runtime just like we do with the DLL, that approach is flawed as well. Many antivirus and malware monitors will happily allow any application to place a DLL in the temp directory and let an executable call LoadLibrary on it, but having an executable file magically appear and attempt to run in the temp directory throws up all kinds of flags. Making it a requirement </span><span style="font-family: "arial" , "helvetica" , sans-serif;">to disable your antimalware software be</span><span style="font-family: "arial" , "helvetica" , sans-serif;">fore using the IE driver is not something I'd be comfortable with.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Creating a second executable, and its attendant DLL for using the hook procedure, <i>and</i> figuring out some way for the two executables to communicate with each other amounts to a massive rearchitecture of the IE driver. With Microsoft's <a href="http://blogs.msdn.com/b/ie/archive/2014/08/07/stay-up-to-date-with-internet-explorer.aspx">announcement</a> of the sunset of support for all legacy versions of IE in January 2016, and with the creation of <a href="http://www.microsoft.com/en-us/download/details.aspx?id=44069">their own WebDriver implementation</a>, it's not clear that the benefit of making these intrusive changes will outweigh the cost of doing so.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span></div>
</div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com6tag:blogger.com,1999:blog-7433739558862985636.post-4518089593492220862014-09-10T10:26:00.000-07:002014-09-12T08:00:50.705-07:00Using the Internet Explorer WebDriver Implementation from Microsoft<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">Microsoft recently delivered an implementation of an Internet Explorer driver. This is great news, and should be a real help to users of the Selenium project. Concurrent with that release, the IEDriverServer.exe has been updated to take advantage of this new implementation.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">The integration with IEDriverServer has been implemented as a new command-line switch on IEDriverServer.exe. By launching IEDriverServer.exe using the <span style="font-family: "Courier New",Courier,monospace;">--implementation=<value></span> switch, you can force the executable to use a specific driver implementation. Valid values for the switch are:</span><br />
<ul style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-family: "Courier New",Courier,monospace;">LEGACY</span> - Uses the existing open-source driver implementation</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-family: "Courier New",Courier,monospace;">VENDOR</span> - Forces the driver to use the Microsoft implementation, regardless of whether prerequisites are met, or whether the installed version of Internet Explorer is the proper version (will throw an exception when creating a new session if the prerequisites are not installed and configured properly)</span></li>
<li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-family: "Courier New",Courier,monospace;">AUTODETECT</span>
- Uses the Microsoft implementation for IE 11, if the prerequisites are
installed, falling back to the open-source implementation if the
components are not present (still under development in the IEDriverServer.exe code)</span></li>
</ul>
<span style="font-family: Arial,Helvetica,sans-serif;">If no value is specified, or if the value passed in is not one of those listed above, IEDriverServer.exe will use the existing open-source implementation. This is only a temporary default; the intent is for the default to shift to be the Microsoft implementation as the specification on which it is based matures.</span><br />
<br />
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Prerequisites</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;">In order to use the Microsoft implementation, you'll need a few prerequisites.</span><br />
<ul style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">You must have the <a href="http://blogs.msdn.com/b/ie/archive/2014/08/13/august-updates-for-internet-explorer.aspx">August 2014 updates to Internet Explorer</a> installed via Windows Update</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">You need to install the <a href="http://www.microsoft.com/en-us/download/details.aspx?id=44069">IE Web Driver Tool for Internet Explorer 11</a></span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">You need <a href="http://selenium-release.storage.googleapis.com/index.html?path=2.43/">version 2.43.0</a> or higher of the language bindings and IEDriverServer.exe</span></li>
</ul>
<br />
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Caveats and Provisos</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;">This integration with IEDriverServer.exe should be considered experimental at the time of this writing. First, please realize that the Microsoft implementation only supports IE11. There are no announced plans for Microsoft to support other versions of Internet Explorer with this WebDriver implementation.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">Also, the Microsoft implementation strictly follows the <a href="https://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html">W3C WebDriver Specification</a>. Since the spec is currently an editors' draft, it does not completely describe the WebDriver API. In other words, there are some features that are implemented in the open-source implementation that are not documented yet in the spec, which in turn means that they are not implemented in the Microsoft implementation.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">Additionally, there are some small differences in the objects sent back and forth across the <a href="https://code.google.com/p/selenium/wiki/JsonWireProtocol">JSON Wire Protocol</a> between the spec and the open-source implementation. The implications are that there may need to be changes made to the individual language bindings to pass the proper JSON payload across to the Microsoft implementation. There is considerable pressure not to update the language bindings, since, again, the spec is currently an editors' draft, and there have been bugs filed against it to have it's protocol more closely match the existing open-source language bindings' implementations.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">In the interest of allowing users to be able to experiment with the Microsoft implementation, the .NET bindings have had all of the necessary protocol changes grafted in, with explicit comments in the source code to have the changes removed when the spec is finalized and all implementations are consistent. The Java bindings have a partial implementation, and can launch IEDriverServer.exe with the proper command-line parameters to enable use of the Microsoft implementation, but the protocol changes have not yet been implemented. Unfortunately, there is no timetable for this work for other language bindings.</span><br />
<br />
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Example</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;">So here's an example of what the code looks like to enable and use the Microsoft implementation of the IE driver, using C# code. It's pretty straightforward.</span><br />
<br />
<pre class="brush: csharp">public static void DriveIEUsingMicrosoftImplementation()
{
InternetExplorerDriverService service =
InternetExplorerDriverService.CreateDefaultService();
service.Implementation = InternetExplorerDriverEngine.Vendor;
IWebDriver driver = new InternetExplorerDriver(service);
} </pre>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">Conversely, one could launch IEDriverServer.exe with the appropriate command-line switch and use <span style="font-family: "Courier New",Courier,monospace;">RemoteWebDriver</span> to talk to that running instance. </span></div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com13tag:blogger.com,1999:blog-7433739558862985636.post-58511247093294497322013-09-16T12:01:00.000-07:002014-09-16T06:45:48.818-07:00Capturing JavaScript Errors in WebDriver - Even on Page Load!<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjKWOEZNUmNp9-TPvsEM5du_CSUD9av-y29DI_tIy6cGcpWRC6MbKCgG3MxnPmDfDkuKlGO6De_hFt5VA5sLYpZnzYDnztfAZ0DE5c8UedAAK5WIjNEGUmxCsLdGKP75V0Nveg_v5ThWI/s1600/proxy.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjKWOEZNUmNp9-TPvsEM5du_CSUD9av-y29DI_tIy6cGcpWRC6MbKCgG3MxnPmDfDkuKlGO6De_hFt5VA5sLYpZnzYDnztfAZ0DE5c8UedAAK5WIjNEGUmxCsLdGKP75V0Nveg_v5ThWI/s320/proxy.jpg" /></a></div>
<br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">A common question I often hear bandied about with WebDriver is, "How can I capture JavaScript errors on the page?" There is an <a href="http://code.google.com/p/selenium/issues/detail?id=148">open issue</a> for this feature in the Selenium issue tracker, but there has been little-to-no development effort expended on solving the problem. One of the major issues is that not all browsers allow WebDriver to hook into the JavaScript execution process in a way that we could retrieve the errors effectively. Internet Explorer is especially bad about this, insisting on not providing COM methods to retrieve the JavaScript errors.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">The most common suggestion for making JavaScript errors available to WebDriver code involves installing an event handler to the onerror event, capturing any uncaught errors to a global variable, and using a script execution to retrieve them. If you have access to the source code of the page you're automating, this is easy, as Alister Scott <a href="http://watirmelon.com/2012/12/19/using-webdriver-to-automatically-check-for-javascript-errors-on-every-page/">has pointed out</a> in the past. In fact, that's the method I'd strongly prefer if I ever need to capture JavaScript errors on a page. However, many people ask about how to do it if they don't have access to modify the source code, and there, </span><span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-family: Arial, Helvetica, sans-serif;">the challenge is that
it's very hard to inject such an event handler early enough in the page
load process to catch errors that may happen in the onload event.</span></span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-family: Arial, Helvetica, sans-serif;">As we learned in <a href="http://jimevansmusic.blogspot.com/2013/08/implementing-webdriver-http-status.html">my previous series</a> on retrieving HTTP response codes, using a proxy is an incredibly powerful way to extend the reach of your WebDriver code, working around things the browser won't, by nature, let you have. With that in mind, I've put together a brief example how to retrieve the JavaScript errors on a page, even those occuring during the onload event. Once again, I'll be using </span></span><span style="font-family: Arial,Helvetica,sans-serif;">Eric Lawrence's (now Telerik's) excellent <a href="http://fiddler2.com/">Fiddler proxy</a>. For the reasons why, you can check out the posts I referred to previously. Also, much of the browser launch code and setup and teardown of the proxy is identical to the previous posts, so I'll omit that for the sake of brevity.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">The typical approach for finding JavaScript errors is a two-phase affair. First, we must inject a script into the page to catch all uncaught JavaScript errors. Such a script usually looks something like this:</span><br />
<pre class="brush: javascript">window.__webdriver_javascript_errors = [];
window.onerror = function(errorMsg, url, lineNumber) {
window.__webdriver_javascript_errors.push(
errorMsg +' (found at ' + url + ', line ' + lineNumber + ')');
};
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">Then, those errors can be retrieved by with WebDriver by using something like this:</span><br />
<pre class="brush: csharp">string errorRetrievalScript =
"return window.__webdriver_javascript_errors;";
IJavaScriptExecutor executor = driver as IJavaScriptExecutor;
ReadOnlyCollection<object> returnedList =
executor.ExecuteScript(errorRetrievalScript)
as ReadOnlyCollection<object>;
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">But let's assume that I have a test page with the following HTML:</span><br />
<pre class="brush: html"><!DOCTYPE html>
<html>
<head>
<title>Page with JavaScript errors on load</title>
<script>
function loadError() {
var xx = document.propertyThatDoesNotExist.xyz;
}
</script>
<head>
<body onload="loadError()">
This page has a JavaScript error in the onload event.
Usually a problem to trap.
</body>
</html>
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">One would expect that, since the aptly-named propertyThatDoesNotExist actually doesn't exist on the document object, a JavaScript error would be produced attempting to access the xyz property. Furthermore, since the function is called during the onload event of this page, the error will occur in that event, and indeed that's what happens. Parenthetically, you can see a <a href="http://the-internet.herokuapp.com/javascript_error">page with exactly this structure</a>, as part of Dave Haeffner's super-cool <a href="http://github.com/arrgyle/the-internet">"The Internet" project</a>, which exists to provide sample pages of "stuff you'll probably run into someday when using WebDriver."</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">So how do we make sure our error-capture script gets injected into the page in time to catch the onload event? Luckily, with a proxy, we can do exactly that. Let's take a look at how we might perform both steps of the process with WebDriver code plus the Fiddler proxy.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Let's start with the navigation portion. Here's the method I created for that:</span>
<br />
<pre class="brush: csharp">public static void NavigateTo(this IWebDriver driver,
string targetUrl)
{
string errorScript =
@"window.__webdriver_javascript_errors = [];
window.onerror = function(errorMsg, url, line) {
window.__webdriver_javascript_errors.push(
errorMsg + ' (found at ' + url + ', line ' + line + ')');
};";
SessionStateHandler beforeRequestHandler =
delegate(Session targetSession)
{
// Tell Fiddler to buffer the response so that we can modify
// it before it gets back to the browser.
targetSession.bBufferResponse = true;
};
SessionStateHandler beforeResponseHandler =
delegate(Session targetSession)
{
if (targetSession.fullUrl == targetUrl &&
targetSession.oResponse
.headers
.ExistsAndContains("Content-Type", "html"))
{
targetSession.utilDecodeResponse();
string responseBody =
targetSession.GetResponseBodyAsString();
string headTag =
Regex.Match(responseBody,
"<head.*>",
RegexOptions.IgnoreCase).ToString();
string addition =
headTag + "<script>" + errorScript + "</script>";
targetSession.utilReplaceOnceInResponse(headTag,
addition,
false);
}
};
FiddlerApplication.BeforeRequest += beforeRequestHandler;
FiddlerApplication.BeforeResponse += beforeResponseHandler;
driver.Url = targetUrl;
FiddlerApplication.BeforeResponse -= beforeResponseHandler;
FiddlerApplication.BeforeRequest -= beforeRequestHandler;
}
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">Looking closely at the code in this method, what we are doing here is attaching event handlers to manipulate the traffic sent over the wire. In the BeforeRequest event handler, we simply tell Fiddler that we want to examine and modify the response before it is sent along to the browser by setting the bBufferResponse property to true. The BeforeResponse event occurs after the response content has been received by the proxy, but before it has been forwarded to the browser. Here, we look for the close of the <head> tag in the response body, and add a <script> tag with our error-handling script immediately following. This ensures that our error-handling script is the first script executed by the browser. Note that this is an extremely crude and naive method of determining where to inject the script tag; in your implementation, you may require something a bit more sophisticated.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Okay, now we have the code in place to capture the errors, we need a method to retrieve them. The earlier fragment gives you the idea how this will look, but here's the more complete version:</span>
<br />
<pre class="brush: csharp">public static IList<string> GetJavaScriptErrors(
this IWebDriver driver, TimeSpan timeout)
{
string errorRetrievalScript =
@"var errorList = window.__webdriver_javascript_errors;
window.__webdriver_javascript_errors = [];
return errorList;";
DateTime endTime = DateTime.Now.Add(timeout);
List<string> errorList = new List<string>();
IJavaScriptExecutor executor = driver as IJavaScriptExecutor;
ReadOnlyCollection<object> returnedList =
executor.ExecuteScript(errorRetrievalScript)
as ReadOnlyCollection<object>;
while (returnedList == null && DateTime.Now < endTime)
{
System.Threading.Thread.Sleep(250);
returnedList =
executor.ExecuteScript(errorRetrievalScript)
as ReadOnlyCollection<object>;
}
if (returnedList == null)
{
return null;
}
else
{
foreach (object returnedError in returnedList)
{
errorList.Add(returnedError.ToString());
}
}
return errorList;
}
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">A few features to note here. First, the retrieval script clears the cached JavaScript errors as it retrieves them. That allows you to use the same technique to check for errors after any particular action that might yield JavaScript errors. Secondly, I've added a timeout to this method, just in case no JavaScript can load on the page for some reason. This will return null, and allow us to distinguish between that error condition and legitimately having no JavaScript errors on the page.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">One further thing you'll notice is that, <a href="http://jimevansmusic.blogspot.com/2013/08/implementing-http-status-codes-in_17.html">as before in other examples</a>, I'm using the "this" keyword as part of the argument for the driver argument. That allows these methods to be seen as .NET <a href="http://msdn.microsoft.com/en-us/library/vstudio/bb383977.aspx">extension methods</a>, making the syntax when using them a little cleaner. All that remains is to put these in action, like this:</span>
<br />
<pre class="brush: csharp">private static void TestJavaScriptErrors(IWebDriver driver)
{
string url = "http://path/to/your/jserror.html";
Console.WriteLine("Navigating to {0}", url);
driver.NavigateTo(url);
IList<string> javaScriptErrors = driver.GetJavaScriptErrors();
if (javaScriptErrors == null)
{
Console.WriteLine("Could not access JavaScript errors.");
}
else
{
if (javaScriptErrors.Count > 0)
{
Console.WriteLine("Found the following JavaScript errors:");
foreach (string javaScriptError in javaScriptErrors)
{
Console.WriteLine(javaScriptError);
}
}
else
{
Console.WriteLine("No JavaScript errors found.");
}
}
}
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">When run against the test page above, the you will receive output similar to the following (specific error text varies from browser-to-browser, output from Internet Explorer is shown):</span><br />
<br />
<pre>Navigating to http://path/to/your/jserror.html
Found the following JavaScript errors:
Unable to get property 'xyz' of undefined or null reference
(found at http://path/to/your/jserror.html, line 7)</pre>
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">As with previous examples featuring proxies, you can see the full example in the <a href="https://github.com/jimevans/WebDriverProxyExamples">GitHub repository</a> for them. This particular example can be seen in the JavaScriptErrorsExample project within that solution. </span>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com1tag:blogger.com,1999:blog-7433739558862985636.post-49240493974338873052013-08-26T09:40:00.000-07:002013-09-12T08:37:53.092-07:00Implementing HTTP Status Codes in WebDriver, Part 3: Fit and Finish<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0HABWZJ2hNbThfomRhyphenhyphenMK6CYumuFiIJMagOsGFtyEFJicHbNSqo9Lal0CaeQJrb7rMXiTumlP7_8-mfYESxdeDuQzS2mkG0rRPTFvDqo9BVAuWRJOP5vX7o5lC2kmZDEL3LXLkGWQa9s/s1600/3vg5xk.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0HABWZJ2hNbThfomRhyphenhyphenMK6CYumuFiIJMagOsGFtyEFJicHbNSqo9Lal0CaeQJrb7rMXiTumlP7_8-mfYESxdeDuQzS2mkG0rRPTFvDqo9BVAuWRJOP5vX7o5lC2kmZDEL3LXLkGWQa9s/s320/3vg5xk.jpg" width="320" /></a></div>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">This is the final part in my blog series about implementing retrieval of HTTP status codes in WebDriver. In <a href="http://jimevansmusic.blogspot.com/2013/08/implementing-webdriver-http-status.html">Part 1</a>, I demonstrated the basic premise of enabling use of a proxy to monitor HTTP traffic between the browser and the server providing the pages. In <a href="http://jimevansmusic.blogspot.com/2013/08/implementing-http-status-codes-in.html">Part 2</a>, I expanded that solution to actually inspect the traffic for the HTTP status codes. In this part, we'll be finishing off the solution by demonstrating how it works cross-browser, and using a few more tweaks to make the solution a little more elegant.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">First, let's tackle the cross-browser cases. We'll start by creating a factory and an enum to smooth the creation of browsers of different types. First the enum:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<pre class="brush: csharp">enum BrowserKind
{
InternetExplorer,
IE = InternetExplorer,
Firefox,
Chrome,
PhantomJS
}
</pre>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Now, let's create the factory method which instantiates the browsers. I'm not showing the class declaration to save space, but I'm creating the factory methods in a static class called WebDriverFactory.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<pre class="brush: csharp">public static IWebDriver CreateWebDriverWithProxy(BrowserKind kind,
Proxy proxy)
{
IWebDriver driver = null;
switch (kind)
{
case BrowserKind.InternetExplorer:
driver = CreateInternetExplorerDriverWithProxy(proxy);
break;
case BrowserKind.Firefox:
driver = CreateFirefoxDriverWithProxy(proxy);
break;
case BrowserKind.Chrome:
driver = CreateChromeDriverWithProxy(proxy);
break;
default:
driver = CreatePhantomJSDriverWithProxy(proxy);
break;
}
return driver;
}
</pre>
<span style="font-family: Arial, Helvetica, sans-serif;">Now, I'll list out each of the driver creation methods. These are pretty self-explanatory, but quirks of each driver are noted in the comments in the source code.</span><br />
<pre class="brush: csharp">private static IWebDriver CreateInternetExplorerDriverWithProxy(Proxy proxy)
{
InternetExplorerOptions ieOptions = new InternetExplorerOptions();
ieOptions.Proxy = proxy;
// Make IE not use the system proxy, and clear its cache before
// launch. This makes the behavior of IE consistent with other
// browsers' behavior.
ieOptions.UsePerProcessProxy = true;
ieOptions.EnsureCleanSession = true;
IWebDriver driver = new InternetExplorerDriver(ieOptions);
return driver;
}
private static IWebDriver CreateFirefoxDriverWithProxy(Proxy proxy)
{
// A future version of the .NET Firefox driver will likely move
// to an "Options" model to be more consistent with other browsers'
// API.
FirefoxProfile profile = new FirefoxProfile();
profile.SetProxyPreferences(proxy);
IWebDriver driver = new FirefoxDriver(profile);
return driver;
}
private static IWebDriver CreateChromeDriverWithProxy(Proxy proxy)
{
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.Proxy = proxy;
IWebDriver driver = new ChromeDriver(chromeOptions);
return driver;
}
private static IWebDriver CreatePhantomJSDriverWithProxy(Proxy proxy)
{
// This is an egregiously inconsistent API. Expect this to change
// so that an actual Proxy object can be passed in.
PhantomJSDriverService service =
PhantomJSDriverService.CreateDefaultService();
service.ProxyType = "http";
service.Proxy = proxy.HttpProxy;
IWebDriver driver = new PhantomJSDriver(service);
return driver;
}
</pre>
<span style="font-family: Arial, Helvetica, sans-serif;">Now that we have the WebDriverFactory class created, we can update our main method to its final form, which is the following:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<pre class="brush: csharp">static void Main(string[] args)
{
// Note that we're using a desired port of 0, which tells
// Fiddler to select a random available port to listen on.
int proxyPort = StartFiddlerProxy(0);
// We are only proxying HTTP traffic, but could just as easily
// proxy HTTPS or FTP traffic.
OpenQA.Selenium.Proxy proxy = new OpenQA.Selenium.Proxy();
proxy.HttpProxy = string.Format("127.0.0.1:{0}", proxyPort);
// You can uncomment any of the lines below to verify that the
// retrieval of HTTP status codes works properly for each browser.
IWebDriver driver = WebDriverFactory.CreateWebDriverWithProxy(BrowserKind.IE, proxy);
//IWebDriver driver = WebDriverFactory.CreateWebDriverWithProxy(BrowserKind.Firefox, proxy);
//IWebDriver driver = WebDriverFactory.CreateWebDriverWithProxy(BrowserKind.Chrome, proxy);
//IWebDriver driver = WebDriverFactory.CreateWebDriverWithProxy(BrowserKind.PhantomJS, proxy);
TestStatusCodes(driver);
driver.Quit();
StopFiddlerProxy();
Console.WriteLine("Complete! Press <Enter> to exit.");
Console.ReadLine();
}
</pre>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">We're pretty much done with our final solution, except for one final tweak. Let's revisit our NavigateTo and ClickNavigate methods from Part 2 which actually retrieve the HTTP status code. Take a look at the signatures of each of those methods:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<pre class="brush: csharp">public static int NavigateTo(IWebDriver driver, string targetUrl)
public static int ClickNavigate(IWebElement element)
</pre>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">One of the super-groovy things about the .NET Framework since version 3.0 is the introduction of <a href="http://msdn.microsoft.com/en-us/library/bb383977%28v=vs.110%29.aspx">extension methods</a>. These allow you to extend a type with methods of your own design, allowing you to write code as if that type had that method to begin with. Our two methods are tailor-made to be used as extension methods. Simply changing the signature to the following will make that work. I'd also recommend moving those methods to a new static class named something like ExtensionMethods for clarity, but that's up to you.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<pre class="brush: csharp">public static int NavigateTo(this IWebDriver driver, string targetUrl)
public static int ClickNavigate(this IWebElement element)
</pre>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">That means that the final version of our TestStatusCodes method looks like this:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<pre class="brush: csharp">private static void TestStatusCodes(IWebDriver driver)
{
// Using Mozilla's main page, because it demonstrates some of
// the potential problems with HTTP status code retrieval, and
// why there is not a one-size-fits-all approach to it.
string url = "http://www.mozilla.org/";
// Note that the standard IWebDriver interface doesn't have
// a NavigateTo() method that takes a URL and returns a status
// code. However, thanks to the magic of extension methods,
// we can make it look like it does, and call it directly off
// the driver object.
int responseCode = driver.NavigateTo(url);
Console.WriteLine("Navigation to {0} returned response code {1}",
url, responseCode);
string elementId = "firefox-promo-link";
// We're using the same extension method magic here to add in
// a ClickNavigate() method which looks like it's directly
// implemented by IWebElement, even though it really isn't.
IWebElement element = driver.FindElement(By.Id(elementId));
responseCode = element.ClickNavigate();
Console.WriteLine("Element click returned response code {0}",
responseCode);
// Demonstrates navigating to a 404 page.
url = "http://www.mozilla.org/en-US/doesnotexist.html";
responseCode = driver.NavigateTo(url);
Console.WriteLine("Navigation to {0} returned response code {1}",
url, responseCode);
}
</pre>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">We'd also probably want to revisit our timeout code in those methods, probably by providing additional overloads that would make it configurable. I've done that in my local version, and it seems to work pretty well. If you want to see all of this code in a single place, you can take a look at the <a href="https://github.com/jimevans/WebDriverProxyExamples">GitHub repository</a> for this and other example projects on using a proxy.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The argument of the WebDriver project committers regarding HTTP status codes is that a method to retrieve them is out of scope for the API. Furthermore, the explanation has been that the proper approach, one that will work for all browsers, without introducing a suboptimal feature to the WebDriver API, is to use a proxy to capture the HTTP traffic and analyze it yourself. The response to that argument has often been that's too hard to do, and it's stupid to use a screwdriver to put in a screw, when one has a hammer that will work just as well. Hopefully, with this series of blog posts, I've shown that it's pretty easy to work out the use of a proxy to get the information you want. My example is in the .NET bindings, but Java, Ruby, and Python examples would look similar, when using a software-based proxy written in those languages.</span>
Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com3tag:blogger.com,1999:blog-7433739558862985636.post-6995070341749702062013-08-13T16:07:00.000-07:002013-08-21T04:00:48.392-07:00Implementing HTTP Status Codes in WebDriver, Part 2: Achievement Unlocked<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkJD3wnXlljE-JYtNHaRN16jVjtrqJZL-yjWZcOOmsFxyAMp_f-Hr3q80wYLzZ05avW4zunqpsEBJaC_b0EhxLkmjWLdY3vOudG0gi0yO1nsH0d8fN0S030uIzEm9YuNZirxupuL6R6so/s1600/achievement.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkJD3wnXlljE-JYtNHaRN16jVjtrqJZL-yjWZcOOmsFxyAMp_f-Hr3q80wYLzZ05avW4zunqpsEBJaC_b0EhxLkmjWLdY3vOudG0gi0yO1nsH0d8fN0S030uIzEm9YuNZirxupuL6R6so/s320/achievement.gif" height="58" width="320" /></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><strong>UPDATE (21 August 2013):</strong> In response to a comment by Eric Lawrence (author of Fiddler and all around awesome chap), I've updated the code sample for the redirect case. Thanks Eric for taking the time to comment and point out where I could make improvements.</span><br />
<span style="font-family: Arial;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">In <a href="http://jimevansmusic.blogspot.com/2013/08/implementing-webdriver-http-status.html">Part 1</a> of this series, we looked at the beginnings of implementing HTTP status codes in WebDriver the correct way. That is to say, by using a proxy server to monitor traffic for the information we want. To recap, we're using <a href="http://fiddler2.com/">Fiddler</a> as our proxy, the <a href="http://code.google.com/p/selenium/downloads/detail?name=selenium-dotnet-2.35.0.zip">.NET bindings</a> to execute our WebDriver code, and we're running against <a href="http://www.mozilla.org/">Mozilla's website</a> as our test destination. At the end of the last blog post, we successfully had a proxy hooked up, which will log resources to the console as they are requested by the browser. Now it's time to actually extract the HTTP status codes from the information that the proxy is able to collect. As a reminder, here's what our WebDriver execution looks like:</span><br />
<pre class="brush: csharp">private static void TestStatusCodes(IWebDriver driver)
{
// Using Mozilla's main page, because it demonstrates some of
// the potential problems with HTTP status code retrieval, and
// why there is not a one-size-fits-all approach to it.
string url = "http://www.mozilla.org/";
driver.Navigate().GoToUrl(url);
string elementId = "firefox-promo-link";
IWebElement element = driver.FindElement(By.Id(elementId));
element.Click();
// Demonstrates navigating to a 404 page.
url = "http://www.mozilla.org/en-US/doesnotexist.html";
driver.Navigate().GoToUrl(url);
}</pre>
<span style="font-family: Arial, Helvetica, sans-serif;">So the first thing we are doing in our WebDriver code is navigating to http://www.mozilla.org/. So let's create a method that will perform the navigation, and return us the status code. As we saw last time, Fiddler lets us hook up an event delegate to respond every time a resource is retrieved by the browser, and analyze that response. The nice thing about event delegates in .NET is that we don't need to leave them hooked up any longer than necessary. Here's our first stab at a method that will hook and unhook the delegate for the navigation:</span><br />
<pre class="brush: csharp">public static int NavigateTo(IWebDriver driver, string targetUrl)
{
int responseCode = 0;
SessionStateHandler responseHandler = delegate(Session targetSession)
{
responseCode = targetSession.responseCode;
};
FiddlerApplication.AfterSessionComplete += responseHandler;
driver.Url = targetUrl;
while (responseCode == 0)
{
System.Threading.Thread.Sleep(100);
}
FiddlerApplication.AfterSessionComplete -= responseHandler;
return responseCode;
}
</pre>
<span style="font-family: Arial, Helvetica, sans-serif;">Astute readers will see that this has a couple of issues with it. First, how do we know what behavior we want for redirects? Our base URL to which we're navigating has just such a redirect. Do we expect to return a 300-level response, or follow the navigations through until we receive a 200-level or 400-level response? This is a perfect example of why there's no one-size-fits-all approach to HTTP status codes that will work for every WebDriver user, and a reason why, in turn, this feature is out of scope in the WebDriver API. In our case, if the URL redirects for navigation, we're going to return the redirect response code. In your implementation, if you decide on another approach, you'll want to modify the event handler delegate to meet your own needs.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">The second issue is that we aren't guaranteed that we're returning the response code for the proper resource. So we want a modification that will validate that. Also, we'll probably want to create a timeout so that we don't inadvertently loop infinitely in the while loop. Making these modifications, you'll get a method that looks something like this:</span><br />
<pre class="brush: csharp">public static int NavigateTo(IWebDriver driver, string targetUrl)
{
int responseCode = 0;
SessionStateHandler responseHandler = delegate(Session targetSession)
{
if (targetSession.fullUrl == targetUrl)
{
responseCode = targetSession.responseCode;
}
};
FiddlerApplication.AfterSessionComplete += responseHandler;
// Yes, we're hard-coding a 10 second timeout here. Don't worry, we'll
// make that configurable before we're done.
DateTime endTime = DateTime.Now.Add(TimeSpan.FromSeconds(10));
driver.Navigate().GoToUrl(targetUrl);
while (responseCode == 0 && DateTime.Now < endTime)
{
System.Threading.Thread.Sleep(100);
}
FiddlerApplication.AfterSessionComplete -= responseHandler;
return responseCode;
}
</pre>
<span style="font-family: Arial, Helvetica, sans-serif;">Okay, so now we have a method that will return us the status code on explicit navigation to a URL. What about on a click that navigates to a new location? Clicks are a little trickier, because a click might trigger a navigation, or it might not. In my opinion, you should know what type of click you'll be performing, so I'll create a method that we will explicitly call when we want to perform a click that will navigate, and return the HTTP status code of that navigation. I'll also take this opportunity to demonstrate a way to handle redirects, since the link we're clicking on in our test code also causes a redirect. Again, we'll hook up a delegate for the duration of the time we need it, and unhook it after we're done.</span><br />
<pre class="brush: csharp">public static int ClickNavigate(IWebElement element)
{
int responseCode = 0;
string targetUrl = string.Empty;
SessionStateHandler responseHandler = delegate(Session targetSession)
{
// For the first session of the click, the URL should be the initial
// URL requested by the element click.
if (string.IsNullOrEmpty(targetUrl))
{
targetUrl = targetSession.fullUrl;
}
// This algorithm could be much more sophisticated based on your
// needs. In our case, we'll only look for responses where the
// content type is HTML, and that the URL of the session matches
// our current target URL. Note that we also only set the response
// code if it's not already been set.
if (targetSession.oResponse["Content-Type"].Contains("text/html") &&
targetSession.fullUrl == targetUrl &&
responseCode == 0)
{
// If the response code is a redirect, get the URL of the
// redirect, so that we can look for the next response from
// the session for that URL.
if (targetSession.responseCode >= 300 &&
targetSession.responseCode < 400)
{
// Use GetRedirectTargetURL rather than examining the
// "Location" header, as some sites (illegally) might
// use a relative URL for the header (per Eric Lawrence).
targetUrl = targetSession.GetRedirectTargetURL();
}
else
{
responseCode = targetSession.responseCode;
}
}
};
// Note that we're using the ResponseHeadersAvailable event so
// as to avoid a race condition with the browser (per Eric
// Lawrence).
FiddlerApplication.ResponseHeadersAvailable += responseHandler;
// Yes, we're hard-coding a 10 second timeout here. Don't worry, we'll
// make that configurable before we're done.
DateTime endTime = DateTime.Now.Add(TimeSpan.FromSeconds(10));
element.Click();
while (responseCode == 0 && DateTime.Now < endTime)
{
System.Threading.Thread.Sleep(100);
}
FiddlerApplication.ResponseHeadersAvailable -= responseHandler;
return responseCode;
}
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">All that remains is to modify our WebDriver code to call our new methods instead of the standard WebDriver ones, and add some console logging to prove that we get actual status codes returned from our methods. That modifies our TestStatusCodes method to look like this:</span><br />
<pre class="brush: csharp">private static void TestStatusCodes(IWebDriver driver)
{
// Using Mozilla's main page, because it demonstrates some of
// the potential problems with HTTP status code retrieval, and
// why there is not a one-size-fits-all approach to it.
string url = "http://www.mozilla.org/";
int responseCode = NavigateTo(driver, url);
Console.WriteLine("Navigation to {0} returned response code {1}",
url, responseCode);
string elementId = "firefox-promo-link";
IWebElement element = driver.FindElement(By.Id(elementId));
responseCode = ClickNavigate(element);
Console.WriteLine("Element click returned response code {0}",
responseCode);
// Demonstrates navigating to a 404 page.
url = "http://www.mozilla.org/en-US/doesnotexist.html";
responseCode = NavigateTo(driver, url);
Console.WriteLine("Navigation to {0} returned response code {1}",
url, responseCode);
}</pre>
<span style="font-family: Arial, Helvetica, sans-serif;">Running our console application from last time, we now will receive output that looks like the following:</span><br />
<pre>Starting Fiddler proxy
Fiddler proxy listening on port 62594
Navigating to http://www.mozilla.org/
Navigation to http://www.mozilla.org/ returned response code 301
Clicking on element with ID firefox-promo-link
Element click returned response code 200
Navigating to http://www.mozilla.org/en-US/doesnotexist.html
Navigation to http://www.mozilla.org/en-US/doesnotexist.html returned response code 404
Shutting down Fiddler proxy
Complete! Press <Enter> to exit.</pre>
<span style="font-family: Arial, Helvetica, sans-serif;">Now we have a fully functioning example for Firefox. Next time, we'll add the code to make it cross-browser aware, and add a few more tricks to make it more elegant for use with WebDriver. </span>Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com9tag:blogger.com,1999:blog-7433739558862985636.post-81287061809346270002013-08-09T08:04:00.000-07:002013-08-17T08:45:22.505-07:00Implementing HTTP Status Codes in WebDriver, Part 1: Challenge Accepted<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmUrl2IAnGdrvXF3-f53EBd2b9sZYuPQg8HuzXRHgNBZvwBxbvXNFJ6EU_hSU77s8SxGqsEqd1JxUSwYOtbEE4NsgOW1gbKCm75Zq5fLIXGlCgf3HoMP-qKgFDpnQhHFjmFrcHhcJzLtk/s1600/challenge.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmUrl2IAnGdrvXF3-f53EBd2b9sZYuPQg8HuzXRHgNBZvwBxbvXNFJ6EU_hSU77s8SxGqsEqd1JxUSwYOtbEE4NsgOW1gbKCm75Zq5fLIXGlCgf3HoMP-qKgFDpnQhHFjmFrcHhcJzLtk/s320/challenge.jpg" width="320" /></a></div>
<span style="font-family: Arial,Helvetica,sans-serif;">A while back, <a href="http://jimevansmusic.blogspot.com/2012/07/webdriver-y-u-no-have-http-status-codes.html">I wrote a post</a> that discussed at length why HTTP status codes are not present in the WebDriver API. Furthermore, the post went on to explain why I believe they're not needed in the API, and that there are other tools better suited to retrieving this particular piece of esoterica. Since I wrote that article, other Selenium contributors <a href="http://blog.rocketpoweredjetpants.com/2013/06/the-unix-philosophy-webdriver-and-http.html">have written</a> <a href="http://www.theautomatedtester.co.uk/blog/2013/the-tale-of-selenium-bug-141.html">about the same topic</a>. The general premise of those blog posts and mine is that using a proxy is the proper way to capture the status code if you actually require it.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Nevertheless, <a href="http://code.google.com/p/selenium/issues/detail?id=141">the issue in the Selenium issue tracker</a> that was the inspiration for those blog posts continues to receive comments, most of them vehemently opposed to the decision of the Selenium development team. The decision has been called "<a href="http://code.google.com/p/selenium/issues/detail?id=141#c62">complete nonsense</a>," "<a href="http://code.google.com/p/selenium/issues/detail?id=141#c87">silly</a>," "<a href="http://code.google.com/p/selenium/issues/detail?id=141#c96">condescending</a>," and "<a href="http://code.google.com/p/selenium/issues/detail?id=141#c104">simply defective</a>," among other choice phrases. My colleagues on the Selenium project have posted code samples that show how to effectively use a proxy with Selenium to solve this problem, and the response to those samples has been that they aren't detailed enough.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Alright, fine. Time to put my proverbial money where my mouth is. I recently looked into what it would take to actually implement a proxy solution, with correct return of HTTP status codes, including writing all of the code necessary to extract it. It doesn't take that much, as it turns out. Once I'd settled on a technology to use, I had a full working example in about 4 hours. Let's take a look at how this would work.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">In my example, I decided to use the Mozilla website, <a href="http://www.mozilla.org/">http://www.mozilla.org/</a>, as my test. I settled on this because the site isn't likely to disappear anytime soon, and as currently written, it nicely demonstrates some of the issues inherent in getting HTTP status codes. Please note that I don't own the website, so it's possible that these examples could break at any time after this writing; at some point, I'll look at creating a standalone website that illustrates the same concepts. I'm also going to be using the WebDriver .NET bindings, and specifically, <a href="http://code.google.com/p/selenium/downloads/detail?name=selenium-dotnet-2.35.0.zip">version 2.35.0</a> of the .NET bindings. For the proxy component, I decided to use Eric Lawrence's (now Telerik's) excellent <a href="http://fiddler2.com/">Fiddler proxy</a>.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Let's talk for a moment about why I chose Fiddler. First, </span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-family: Arial,Helvetica,sans-serif;">I'm a .NET guy,
and I try to look for solutions that don't require another runtime (like
Java, Ruby, or Python) if possible. </span>Second, Fiddler offers me the <a href="http://fiddler2.com/fiddlercore">FiddlerCore component</a>, which allows me to use an API-only version of Fiddler, and programmatic access to all of the proxy's settings and data. The API may be a little less polished than other .NET component APIs, but it does use an event-driven model, which appeals to me as a .NET developer. While Fiddler isn't open-source, it is free to use, with no feature restrictions based on free vs. paid use. With all of that in mind, let's begin. Here's the basic code that I want to run, using the standard WebDriver API:</span><br />
<pre class="brush: csharp">private static void TestStatusCodes(IWebDriver driver)
{
// Using Mozilla's main page, because it demonstrates some of
// the potential problems with HTTP status code retrieval, and
// why there is not a one-size-fits-all approach to it.
string url = "http://www.mozilla.org/";
driver.Navigate().GoToUrl(url);
string elementId = "firefox-promo-link";
IWebElement element = driver.FindElement(By.Id(elementId));
element.Click();
// Demonstrates navigating to a 404 page.
url = "http://www.mozilla.org/en-US/doesnotexist.html";
driver.Navigate().GoToUrl(url);
}
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">I'll be running this method from within a console application, with the main method looking something like this:</span><br />
<pre class="brush: csharp">static void Main(string[] args)
{
// Eventually, we will use different browsers to prove this
// solution works cross-browser, but for now, we will use
// Firefox only.
IWebDriver driver = new FirefoxDriver();
TestStatusCodes(driver);
driver.Quit();
Console.WriteLine("Complete! Press <Enter> to exit.");
Console.ReadLine();
}
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">Let's look at how to integrate Fiddler in this solution. Starting the proxy server couldn't be easier. We'll create a method to do this. One thing to note in the method is that we can either specify a port for the proxy to listen on, or let Fiddler pick one for us.</span><br />
<pre class="brush: csharp">private static int StartFiddlerProxy(int desiredPort)
{
// We explicitly do *NOT* want to register this running Fiddler
// instance as the system proxy. This lets us keep isolation.
Console.WriteLine("Starting Fiddler proxy");
FiddlerCoreStartupFlags flags = FiddlerCoreStartupFlags.Default &
~FiddlerCoreStartupFlags.RegisterAsSystemProxy;
FiddlerApplication.Startup(desiredPort, flags);
int proxyPort = FiddlerApplication.oProxy.ListenPort;
Console.WriteLine("Fiddler proxy listening on port {0}", proxyPort);
return proxyPort;
}</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">Technically speaking, we probably don't need to shut down the proxy, since it's the last thing we do before our main method exits, but we're going to be a good citizen and shut it down anyway.</span><span style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<pre class="brush: csharp">private static int StopFiddlerProxy()
{
Console.WriteLine("Shutting down Fiddler proxy");
FiddlerApplication.Shutdown();
}</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">All that remains is to hook up an event handler so that we can analyze the traffic that comes through the proxy, and to make the Firefox driver aware of the proxy. We can do those things within the context of our main method. After all of these, the final main method looks like this:</span><br />
<pre class="brush: csharp">static void Main(string[] args)
{
// Note that we're using a desired port of 0, which tells
// Fiddler to select a random available port to listen on.
int proxyPort = StartFiddlerProxy(0);
// Hook up the event for monitoring proxied traffic.
FiddlerApplication.AfterSessionComplete += delegate(Session targetSession)
{
Console.WriteLine("Requested resource from URL {0}",
targetSession.fullUrl);
};
// We are only proxying HTTP traffic, but could just as easily
// proxy HTTPS or FTP traffic.
OpenQA.Selenium.Proxy proxy = new OpenQA.Selenium.Proxy();
proxy.HttpProxy = string.Format("127.0.0.1:{0}", proxyPort);
// Eventually, we will use different browsers to prove this
// solution works cross-browser, but for now, we will use
// Firefox only.
FirefoxProfile profile = new FirefoxProfile();
profile.SetProxyPreferences(proxy);
IWebDriver driver = new FirefoxDriver(profile);
TestStatusCodes(driver);
driver.Quit();
Console.WriteLine("Complete! Press <Enter> to exit.");
Console.ReadLine();
}
</pre>
<span style="font-family: Arial,Helvetica,sans-serif;">When we run our console application, we see something like this:</span><br />
<pre>Starting Fiddler proxy
Fiddler proxy listening on port 62492
Navigating to http://www.mozilla.org/
Requested resource from URL http://www.mozilla.org/
Requested resource from URL http://mozorg.cdn.mozilla.net/media/css/tabzilla-min.css?build=c2a3f7a
Requested resource from URL http://mozorg.cdn.mozilla.net/media/js/site-min.js?build=c2a3f7a
Requested resource from URL http://mozorg.cdn.mozilla.net/media/css/responsive-min.css?build=c2a3f7a
Requested resource from URL http://mozorg.cdn.mozilla.net/media/img/favicon.ico
Requested resource from URL http://www.mozilla.org/en-US/
[...
Many resources deleted for brevity
...]
Clicking on element with ID firefox-promo-link
Requested resource from URL http://mozorg.cdn.mozilla.net/media/fonts/Vollkorn-Regular-webfont.woff
Requested resource from URL http://mozorg.cdn.mozilla.net/media/fonts/Vollkorn-Bold-webfont.woff
Requested resource from URL http://mozorg.cdn.mozilla.net/media/img/home/promo/flicks/760.jpg
Requested resource from URL http://mozorg.cdn.mozilla.net/media/img/home/promo/android/760.jpg?2013-06
Requested resource from URL http://mozorg.cdn.mozilla.net/media/img/home/promo/makerparty/760.jpg
Navigating to http://www.mozilla.org/en-US/doesnotexist.html
Requested resource from URL http://www.mozilla.org/firefox/
Requested resource from URL http://www.mozilla.org/en-US/firefox/
Requested resource from URL http://mozorg.cdn.mozilla.net/media/css/firefox_fx-min.css?build=c2a3f7a
Requested resource from URL http://mozorg.cdn.mozilla.net/media/img/firefox/template/header-logo.png?2013-06
Requested resource from URL http://www.mozilla.org/en-US/firefox/fx/
[...
Many resources deleted for brevity
...]
Shutting down Fiddler proxy
Complete! Press <Enter> to exit. </pre>
<span style="font-family: Arial,Helvetica,sans-serif;">Obviously, this particular example doesn't get us to our desired goal just yet. However, it does allow us to hook up a proxy. Next time, I'll show you how we can refine this solution to actually extract those status codes.</span>Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com6tag:blogger.com,1999:blog-7433739558862985636.post-61522435983423221402013-01-09T09:29:00.000-08:002013-02-01T10:54:13.765-08:00Revisiting Native Events in the IE Driver<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6-AxANZp9ggUL91na0zxQrOElpMzLwitscbxhvwt5OZMBw6CMMRsE5CP_Vs5AWjty5LnUDN8-OgPtx81sz5c5B8sf1NFEa8vyCAJ-oIk64aiIg9JeYNhjLIhsNjvsIpL24HmrrN6yMvQ/s1600/33102987.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6-AxANZp9ggUL91na0zxQrOElpMzLwitscbxhvwt5OZMBw6CMMRsE5CP_Vs5AWjty5LnUDN8-OgPtx81sz5c5B8sf1NFEa8vyCAJ-oIk64aiIg9JeYNhjLIhsNjvsIpL24HmrrN6yMvQ/s1600/33102987.jpg" height="180" width="320" /></a></div>
<span style="font-family: Arial,Helvetica,sans-serif;">A little over six months ago, I wrote a <a href="http://jimevansmusic.blogspot.com/2012/06/whats-wrong-with-internet-explorer.html">blog post</a> that concentrated on so-called "native events" in the IE driver. </span><span style="font-family: Arial,Helvetica,sans-serif;">To refresh your
memory, native events are using OS-level mechanisms for simulating mouse
and keyboard input in the browser. This is distinct from "simulated
events", which rely on using JavaScript to trigger the events on the
elements in a page. </span><span style="font-family: Arial,Helvetica,sans-serif;">Since I wrote that blog post, there have been some developments in the IE driver that have changed the landscape a little, and it's probably a good time to revisit the topic.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">If you'll recall, there were two major issues that people have had challenges with using native events with the IE driver. The first is mouse clicks being swallowed up if the IE window isn't the foreground window. The second is hovering over elements, where the elements displayed on hover would appear for a fraction of a second, then disappear. This second issue was particularly vexing, since it only happens when the physical mouse cursor is within the bounds of the IE window. See the aforementioned previous blog post if you want more details about what exactly causes these issues, and why they're particularly hard to solve. Having said that, let's look at the newer approaches that have come available in the last six months.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<h4>
<span style="font-family: Arial,Helvetica,sans-serif;">Persistent Hovers</span><span style="font-family: Arial,Helvetica,sans-serif;"> </span></h4>
<span style="font-family: Arial,Helvetica,sans-serif;">The first approach is using what's called "persistent hovers." By default, the IE driver simulates native mouse movements by using the </span><span style="font-family: Arial,Helvetica,sans-serif;">Windows </span><span style="font-family: Arial,Helvetica,sans-serif;"><a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms644950%28v=vs.85%29.aspx">SendMessage API</a> to send <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms645616%28v=vs.85%29.aspx">WM_MOUSEMOVE messages</a> to the IE window. This can work, but if the IE window is in the foreground, and the physical mouse cursor is within the bounds of the IE window, the hover won't persist; it'll flash, and disappear. To solve this, the IE driver launches a separate thread and continually throws WM_MOUSEMOVE messages at the IE window for the coordinates that your test code specifies.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">This means that the hover doesn't flash once and disappear, now it flickers, but at least it sticks around long enough to do something with. With the flickering, there's a far better chance that the hidden element you're attempting to display with your hover will be visible when you try to interact with it. Careful readers will note that there's still a race condition here, which your code might lose. Also, using persistent hovers renders that particular instance of IE unusable if you want to do something with it manually after you've finished with your test. Finally, notice that they only attempt to address the hover problem, not the "no focus click swallowing" problem.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Persistent hovers are available now with native events in the IE driver. They're enabled by default. If, for whatever reason, you need to revert to the behavior without them, they can be controlled by the "enablePersistentHover" capability.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<h4>
<span style="font-family: Arial,Helvetica,sans-serif;">Simulated Events</span></h4>
<span style="font-family: Arial,Helvetica,sans-serif;">Over the last six months or so, there have been some strong improvements made in the simulated events implementation in IE. Simulated events have the advantage that they're not reliant on window focus. They're also a bit faster than the default native events implementation. However, these advantages are not without their costs.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Simulated events are limited to the JavaScript sandbox. There are certain effects that can't be activated via JavaScript, like a <a href="https://developer.mozilla.org/en-US/docs/CSS/:hover">hover triggered solely by CSS</a>. Blocking JavaScript events, like using alert(), confirm() or prompt(), especially in an onclick or onsubmit event handler, can cause your JavaScript execution (and therefore your WebDriver code) to hang.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">If your application doesn't run afoul of the issues endemic to pure JavaScript input simulation, simulated events may be a good choice for you. You can use simulated events by setting the "nativeEvents" capability to false.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<h4>
<span style="font-family: Arial,Helvetica,sans-serif;">Requiring Window Focus</span></h4>
<span style="font-family: Arial,Helvetica,sans-serif;">One of the guiding principles of the WebDriver project is that a driver should not require the browser window to be in focus to work properly. This principle is so important to the project that it's been codified as part of the <a href="https://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html#running-without-window-focus">draft W3C specification</a> for WebDriver. However, there is some subset of users that don't care about needing window focus in their environment, they just want the most accurate test utility possible. Often these people mention that they're running their tests in a test lab or on dedicated virtual machines, and don't need to worry that focus is taken by the browser window. Until now, the IE driver had no way to accommodate such a requirement.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">However, with a <a href="http://code.google.com/p/selenium/source/detail?r=bd02fa0e2059df544eac638a3765df4555717b27">recent change set</a> in the IE driver code base, it is now possible to ask the IE driver to bring the window currently focused in the driver (as determined by the <a href="http://selenium.googlecode.com/svn/trunk/docs/api/java/org/openqa/selenium/WebDriver.html#getWindowHandle()">WebDriver.getWindowHandle</a>) to be the window in the foreground. This attempt is not guaranteed to succeed, because making a window from another process the foreground window is a <a href="http://blogs.msdn.com/b/oldnewthing/archive/2009/02/20/9435239.aspx">Bad Thing</a>, but it's the only way to make simulated input work properly. Additionally, instead of using the <a href="http://blogs.msdn.com/b/oldnewthing/archive/2005/05/30/423202.aspx">flawed SendMessage/PostMessage approach</a>, this change introduces use of the Windows <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms646310%28v=vs.85%29.aspx">SendInput API</a>.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">This functionality is brand new. It should be considered very experimental. All input using the WebDriver Advanced User Interactions API (the "Actions" class) will go through this code path if set correctly, but mouse clicks using <a href="http://selenium.googlecode.com/svn/trunk/docs/api/java/org/openqa/selenium/WebElement.html#click()">WebElement.click()</a> will not yet do so. Additionally, there's no mechanism (<a href="https://groups.google.com/d/topic/selenium-developers/vz5gcGoYNbA/discussion">yet!</a>) in the wire protocol to enforce that a sequence of actions should be handled as an atomic unit, though the new internal classes added to the IE driver will make it easy to implement such a feature. To enable this functionality, set the "requireWindowFocus" capability to true; it is defaulted to false, to maintain existing functionality. Note that requireWindowFocus and enablePersistentHover are mutually exclusive.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Hopefully, these methods will give you the most flexibility with how you perform keyboard and mouse interaction with Internet Explorer. Most of these are hacks at best, but that's the best we can do until Microsoft begins producing the IE driver. </span>Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com7tag:blogger.com,1999:blog-7433739558862985636.post-61423926993807751472012-12-24T08:03:00.001-08:002012-12-24T08:11:50.229-08:00Seeing "Info" Messages in a Log Does Not Automatically Imply "Error"<span style="font-family: Arial, Helvetica, sans-serif;">Here's what seems to be a common question among Java language bindings users of the IE driver (<a href="#meaning">tl;dr</a>):</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;">Whenever I use the IE driver, I see an error in my log. The log output is below. How do I stop it?</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">Started InternetExplorerDriver server (32-bit)</span><br />
<span style="font-family: Courier New, Courier, monospace;">2.28.0.0<br />Listening on port 48630<br />Sep 23, 2012 9:12:21 AM org.apache.http.impl.client.<wbr></wbr>DefaultRequestDirector tryExecute<br />INFO: I/O exception (org.apache.http.<wbr></wbr>NoHttpResponseException) caught when processing request: The target server failed to respond<br />Sep 23, 2012 9:12:21 AM org.apache.http.impl.client.<wbr></wbr>DefaultRequestDirector tryExecute<br />INFO: Retrying request</span></blockquote>
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">Follow-up questions like, "What happens after you receive this message?" or, "Does the rest of your code run?" usually get a response like, "No, the rest of my code works fine with IE, I just see this error message, and I don't think I should be seeing any errors."</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">There are two answers to this question, a short one and a long one. The short one is, "Read the log message. It's clearly tagged as 'INFO', as in an informational message, and not indicative of any problem with the code?" I find that this question often comes from users of Eclipse, and that the Eclipse console has colored the message red, and people are so conditioned to see "red == bad" that they react to the format of the message rather than the content. The content of the message is flagged at a level that means, "Hey, nothing is wrong, we're just telling you about it."</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">If all you care about is how to get around the message, you can stop reading here. The longer answer to the question involves an explanation of why you might receive this message. The explanation requires a little knowledge of the architecture of the WebDriver language bindings in general, and the IE driver in particular. Remember that in the language bindings for most browsers, communication with the browser happens via a "server component", and the language bindings use a <a href="http://code.google.com/p/selenium/wiki/JsonWireProtocol">JSON-over-HTTP protocol</a> to communicate with that server component. The server component could be an instance of a remote WebDriver server (whether <a href="http://code.google.com/p/selenium/downloads/detail?name=selenium-server-standalone-2.28.0.jar">written in Java</a>, or <a href="http://code.google.com/p/strontium">another language</a>); or it could be the Firefox browser extension that the FirefoxDriver installs into an anonymous profile and uses; or it could be an instance of <a href="http://code.google.com/p/chromedriver/downloads/list">chromedriver.exe</a>; or, and this is the important bit for our purposes here, it could be an instance of <a href="http://code.google.com/p/selenium/downloads/detail?name=IEDriverServer_Win32_2.28.0.zip">IEDriverServer.exe</a>. Each of these server components starts up an HTTP server with which the language bindings can communicate.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Let's think about what happens when you ask the Java language bindings to create a new instance of the InternetExplorerDriver class.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Locate the IEDriverServer.exe, either from the system PATH or from a Java system property</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Launch IEDriverServer.exe, including its internal HTTP server</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Use the <a href="http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/index.html">org.apache.http.client.HttpClient class</a> to send HTTP requests to the IEDriverServer's internal HTTP server to launch IE and establish a WebDriver session</span></li>
</ol>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Now looking at those steps, can you see the potential flaw in the execution? If you said "race condition," you earn a gold star. The Java method for launching a process will return before the IEDriverServer's HTTP server has had a chance to fully initialize itself and be ready to receive HTTP requests. However, since IEDriverServer.exe is running in a separate process, there's no way for the Java language bindings to know when the HTTP server has fully initialized. The only way to handle the situation is to poll for the HTTP server, repeatedly sending requests until you get a valid response, and that's the message you're seeing in the log. All the bindings are telling you is that the HTTP server hasn't yet been fully initialized, so it's trying the request again, nothing to see here, move along.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><a name="meaning"></a>The bottom line is that the message is informational, not indicative of an error, or else it would have a different logging level. Remember, the format of a log message is incidental; the important thing is its content.</span>Jim Evanshttp://www.blogger.com/profile/16744470953840993252noreply@blogger.com0