Monday, August 20, 2012

You're Doing It Wrong: IE Protected Mode and WebDriver

There's a common problem most people run into with the Internet Explorer driver when they first start using it with IE 7 and above. Most people start by writing code that looks something like this, expecting it to work on a clean installation of Windows, or at least one with the default settings for Internet Explorer:
WebDriver driver = new InternetExplorerDriver();
driver.get("http://seleniumhq.org");
driver.quit();
Imagine their surprise when they get an exception that looks something like this:
org.openqa.selenium.WebDriverException: Unexpected error launching Internet Explorer. Protected Mode must be set to the same value (enabled or disabled) for all zones. (WARNING: The server did not provide any stacktrace information)
A careful reading of the exception's message tells one exactly what the problem is. People who don't bother to read the exception message then turn to their favorite search engine, and after a quick search, they often blindly modify their code to do something like the following:

DesiredCapabilities caps = DesiredCapabilities.internetExplorer();
caps.setCapability(
    InternetExplorerDriver.INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS,
    true);
WebDriver driver = new InternetExplorerDriver(caps);
driver.get("http://seleniumhq.org");
driver.quit();

While this will certainly get them past the initial exception, and will allow the test to run in most cases without incident, it's patently the Wrong Thing to do. The operative questions then are, "Why is it wrong, and what is the right way?" If you don't care about why it's wrong, and just want to know how to fix it correctly, you can skip the historical background by clicking on this handy tl;dr link.

Why does the IE driver require Protected Mode settings changes anyway?


Way back through the mists of time, before 2006, life was easy for automating Internet Explorer. A browser session was represented by a single instance of the iexplore.exe executable. A framework for driving IE could instantiate the browser as a COM object using CoCreateInstance(), or could easily get the COM interfaces to a running instance by using the presence of ActiveAccessibility and sending a WM_HTML_GETOBJECT message to the appropriate IE window handle. Once the framework had a pointer to the COM interfaces, you could be sure that they'd be valid for the lifetime of the browser. It also meant you could easily attach to the events fired by the browser through the DWebBrowserEvents2 COM interface.

Then along came the combination of IE 7 and Windows Vista. In and effort to reduce the attack surface presented by malicious web sites, IE 7 introduced something called Protected Mode, which leveraged Mandatory Integrity Control in Windows Vista to prevent actions initiated IE, usually initiated by JavaScript, from being able to access the operating system the way it could in prior releases. While this was generally a welcome development for most users of IE, it created all manner of problems for automating IE.

When you cross into or out of Protected Mode by, say, navigating from an internal intranet website to one on the internet, IE has to create a new process, because it can't change the Mandatory Integrity Control level of the existing process. Moreover, in IE versions after 7, it's not always obvious that a Protected Mode boundary has been crossed, since IE tries to present a better user experience by seamlessly merging the browser window of the new process with the already opened browser window. This under-the-covers process switching also means that any references pointing to IE's COM objects before the Protected Mode boundary crossing are left pointing to objects that are no longer used by IE after the boundary crossing.

How do I fix it?


One way to solve the problem is to either turn User Access Control (UAC) off, or elevate your privileges to an administrator when running WebDriver code, because all process started by elevated users have a High Mandatory Integrity Control level. Elevating to administrative privileges was not a great option, because the process of elevating couldn't and shouldn't be automated. Similarly, turning off UAC is unacceptable as it leaves the machine in a vulnerable state. Nevertheless, in pre-2.0 versions of the IE driver, that's exactly what one had to do to get code working, and it was one of the primary motivations for rewriting the IE driver in 2010.

Since the tricky bit to solve is when Protected Mode boundaries are crossed, a design decision was made to eliminate the boundary crossings. The simplest way to do that is to change the Protected Mode settings in the browser to be the same, either enabled or disabled, it doesn't matter which, for all zones. That way, it doesn't matter what navigation occurs, it won't cross the Protected Mode boundary, and won't trigger orphaning of the COM objects the IE driver relies on. Moreover, setting the Protected Mode boundaries for the security zones are per-user settings in Windows, and don't generally require elevated privileges to set them.

So what's with the Capabilities hack?


When the rewritten IE driver was first introduced, it was decided that it would enforce its required Protected Mode settings, and throw an exception if they were not properly set. Protected Mode settings, like almost all other settings of IE, are stored in the Windows registry, and are checked when the browser is instantiated. However, some misguided IT departments make it impossible for developers and testers to set even the most basic settings on their machines.

The driver needed a workaround for people who couldn't set those IE settings because their machine was overly locked down. That's what the capability setting is intended to be used for. It simply bypasses the registry check. Using the capability doesn't solve the underlying problem though. If a Protected Mode boundary is crossed, very unexpected behavior including hangs, element location not working, and clicks not being propagated, could result. To help warn people of this potential problem, the capability was given big scary-sounding names like INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS in Java and IntroduceInstabilityByIgnoringProtectedModeSettings in .NET. We really thought that telling the user that using this setting would introduce potential badness in their code would discourage its use, but it turned out not to be so.

Let me state this now in very clear and unambiguous terms. If you are able to set the Protected Mode settings of IE, and you are still using the capability you are risking the stability of your code. Don't do it. Set the settings. It's not that hard.

How to set Protected Mode settings


In IE, from the Tools menu (or the gear icon in the toolbar in later versions), select "Internet options." Go to the Security tab. At the bottom of the dialog for each zone, you should see a check box labeled "Enable Protected Mode." Set the value of the check box to the same value, either checked or unchecked, for each zone. Here's the dialog for reference:


Note that you don't have to change the slider for security level, and you don't have to disable Protected Mode. I routinely run with Protected Mode turned on for all zones, as I think it provides a more secure browsing experience.


38 comments:

  1. Whoa !! I never knew this nor would I ever have.. I think I will going forward STOP recommending to folks on the Selenium Users forum to use that scary setting "INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS" but instead just on a one time basis try and change the Protected mode as how you have recommended.

    Your blog is definitely a good place to hang around Jim..

    ~Krishnan

    ReplyDelete
  2. Need help. I cannot change the settings for Protected Mode. As its controlled by group policy.

    So i am running tests on a trusted site for which Protected Mode is set to OFF. And after launching the URL, nothing is happening.
    It says No window found exception.

    Setting "INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS" to true also doesn't help :(

    ReplyDelete
    Replies
    1. Sadly, this isn't going to be your best venue for getting help on something like this. I would recommend posting to the WebDriver user mailing list (https://groups.google.com/forum/?fromgroups#!forum/webdriver). Also, when you do so, please be ready to be asked for an HTML page or a URL to a public site that demonstrates the problem. What's that? You can't provide an HTML page? It'll be a lot harder to help you then. (http://jimevansmusic.blogspot.com/2012/12/not-providing-html-page-is-bogus.html)

      Delete
  3. "This under-the-covers process switching also means that any references pointing to IE's COM objects before the Protected Mode boundary crossing are left pointing to objects that are no longer used by IE after the boundary crossing."

    For what its worth, IE8+ fire a "New Process" event when a new tab process is created to allow automation to follow along. http://msdn.microsoft.com/en-us/library/cc288084(v=vs.85).aspx

    ReplyDelete
    Replies
    1. Thanks for the comment Eric. If anyone would have insights into better ways of making this work, it would be you. :) Nevertheless, I've played around with the NewProcess event, but I must be doing something wrong. According to the documentation, there should be a passed-in reference to "the new process object", but every time I see this event fired, that pointer is null.

      Also, multiple instances of the IE driver may be driving multiple instances of IE, not to mention that the user may have his or her own instance open. Given that, there's no rational way I can see to poll for the new process to know *which* one contains the new browser object that *this* particular instance of IE just created.

      Finally, the event is not available in IE7, which the driver has to continue to support. I could work around that with version checking, but I can't seem to get it to work for me.

      Delete
  4. Hi, I currently working with Selenium and good lord, I watched your video and read this log just in time before I head slam into weird IEDriver behavior. Thank you, a lot.
    But, while running my test, with protected mode on for all zones, I noticed that there's a pop up say "Only secure content is displayed". So is that expected or I'm doing something wrong? And if it's expected, than what would I lose if some of my webpage contents are non-secure?

    ReplyDelete
  5. I set the settings as per suggested by you in this blog but doesn't seems to be working for me.Getting below error
    INFO: I/O exception (java.net.SocketException) caught when processing request: Software caused connection abort: recv failed
    Oct 31, 2013 12:58:50 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
    INFO: Retrying request
    org.openqa.selenium.TimeoutException: Timed out after 10 seconds waiting for presence of element located by: By.id: txtHQLoginName

    Please help
    Thanks in Advance

    ReplyDelete
  6. This is a great explanation of why using the FLAKINESS capability is not desired. I have asked our QA multiple times not to use this and manually set the levels on par. I hope this article will breathe some sense.

    ReplyDelete
  7. Hi

    I am working with Selenium and trying tpo handle a Secondary window which opens up after clicking a link on the primary window. The problem is that while executing my scripts the secondary window doesnot open. THe click on the Primary window is un noticed

    Also I am not able to change the settings on IE8 (sine I don't have rights). Can this be handled without changing the settings on IE

    Please help

    Seema

    ReplyDelete
  8. Do the below process.

    import org.openqa.selenium.ie.InternetExplorerDriver;
    import org.openqa.selenium.remote.DesiredCapabilities;

    if (browserName.equalsIgnoreCase("InternetExplorer")) {

    DesiredCapabilities caps = DesiredCapabilities.internetExplorer();

    System.setProperty("webdriver.ie.driver", "drivers/IEDriverServer.exe");
    caps.setCapability( InternetExplorerDriver.INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS,
    true);

    caps.setCapability("nativeEvents", false);
    browser = new InternetExplorerDriver(caps);

    Then after, In IE, from the Tools menu (or the gear icon in the toolbar in later versions), select "Internet options." Go to the Security tab. At the bottom of the dialog for each zone, you should see a check box labeled "Enable Protected Mode." Set the value of the check box to the same value,
    either checked or unchecked, for each zone.

    I have applied the same thing at my end, it works fine.

    ReplyDelete
    Replies
    1. Did you actually read the blog you commented on?

      Delete
    2. Thanks a lot Amit, your solution perfectly worked for me, thanks a lot.

      Delete
  9. Hello all,

    I just fixed this issue in my PC .. i m using Windows 8.1.. My code is as below:

    case "ie":
    System.setProperty("webdriver.ie.driver",
    "D:/Prathima/Selenium/IEDriverServer.exe");
    driver = new InternetExplorerDriver();
    openWebPage();
    break;

    even I got same errors.. but I fixed them by setting Enable protected mode option in all levels. Please follow below steps:

    1. Open Internet Explorer --> Select Internet Options
    2. Select Security Tab --> Select or Tick Enable Protected Mode option -- > Now Select Custom Level button --> Set all the fields to Enable (expect few which were already set to recommended) --> Select Ok/Apply button
    3. Select Security Tab --> Select Local Internet Option --> Select or Tick Enable Protected Mode option
    4. Select Security Tab --> Select Trusted sites Option --> Select or Tick Enable Protected Mode option
    5. Select Security Tab --> Select Restricted Site Option --> Select or Tick Enable Protected Mode option

    Now select Ok and Select apply button and close the internet Explorer and run your code.. It will work as you expected.

    Wish this works for you...

    ReplyDelete
    Replies
    1. Thanks Prathima, it worked perfectly.
      Recommend this easy solution to all

      Delete
    2. Yes this worked for me as well! Thank you!

      Delete
  10. Thanks for posting this. I didn't realize that was the behavior of IE.

    ReplyDelete
  11. Hello everyone. I tried everything but when my system is executing the steps, in some point the system clicks in a link (and open a new page) and we need to login again (and that isn't the expected behaviour). I don't know why the system shows the login page to this new window opened.
    Can you help me? I appreciate any answer that could help me with this problem.

    Have a excellent day.

    ReplyDelete
  12. Use ->

    ignore_mode = opts.delete(:introduce_flakiness_by_ignoring_security_domains) != false

    Goto -> External Libraries- selenium-webdriver - lib - selenium - webdriver - ie - bridge.rb

    Update module IE -> def initialize

    It contains -

    ignore_mode = opts.delete(:introduce_flakiness_by_ignoring_security_domains)

    just add != false so that it becomes ->

    ignore_mode = opts.delete(:introduce_flakiness_by_ignoring_security_domains) != false

    ReplyDelete
    Replies
    1. IF I click on one link it should go to other window rather going there it is asking authentication or login page details. Even we handle both.can u explain me why this happens in IE.

      Delete
  13. I am supposed to drive Internet Explorer on remote VM, so I cannot change the browser setting as the machine provided during run-time will be unknown. Also, I will not be having access to set browser settings manually. Please advice how this situation can be handled.

    ReplyDelete
  14. Started InternetExplorerDriver server (64-bit)
    2.48.0.0
    Listening on port 11255
    Exception in thread "main" org.openqa.selenium.remote.SessionNotFoundException: Unexpected error launching Internet Explorer. Protected Mode settings are not the same for all zones. Enable Protected Mode must be set to the same value (enabled or disabled) for all zones. (WARNING: The server did not provide any stacktrace information)
    Command duration or timeout: 1.32 seconds
    Build info: version: '2.45.0', revision: '5017cb8', time: '2015-02-26 23:59:50'
    System info: host: 'Admin-PC', ip: '192.168.1.5', os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.8.0_45'
    Driver info: org.openqa.selenium.ie.InternetExplorerDriver
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:204)
    at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:162)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:599)
    at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:240)
    at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:225)
    at org.openqa.selenium.ie.InternetExplorerDriver.run(InternetExplorerDriver.java:182)
    at org.openqa.selenium.ie.InternetExplorerDriver.(InternetExplorerDriver.java:174)
    at org.openqa.selenium.ie.InternetExplorerDriver.(InternetExplorerDriver.java:146)
    at Pack12.Project12.main(Project12.java:12)



    while launching ie 11 am getting this error please anyone please help me to resolve this issue.

    Thanks
    Prathap

    ReplyDelete
    Replies
    1. caps.setCapability("ignoreZoomSetting", true);
      driver.findElement(By.tagName("html")).sendKeys(Keys.chord(Keys.CONTROL, "0"));
      I have used this and working fine for me.

      Delete
  15. can we catch org.openqa.selenium.remote.SessionNotFoundException in the sophisticated manner. what is the hierarchy of the this exceptions

    ReplyDelete
  16. IF I click on one link it should go to other window rather going there it is asking authentication or login page details. Even we handle both.can u explain me why this happens in IE.

    ReplyDelete
  17. Hmm.. nice to read but it doesn't work... only option for me is set INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS from Aladeen to Aladeen...

    ReplyDelete
  18. How to open IE Browser in private mode:

    ReplyDelete
  19. Has this been fixed in IE? For myself the corporate policy applied to IE is mixed, and can not be changed(Any changes made are automatically reset). Its setup with one site Checked and Disabled, and unchecked enabled on another.
    Tested on a non Corp PC things work great as explain, but how about when the settings are not the same?

    ReplyDelete
    Replies
    1. Not only *hasn't* it been fixed, it *can't* be fixed. There's no way around the architectural limitations Microsoft has placed in the way. Perhaps in a few weeks, once IE 7 and 8 are no longer under active support, we may be able to revisit the issue.

      Delete
  20. Wow, you come across really jaded in your articles and comments, such a same as the advice you are giving is great, very helpful. I stumbled across here whilst looking for a solution to a window not appearing in the return from getWindowHandles. As part of my search I found this tidbit from yourself "and (2) not bothering to set the Protected Mode settings properly" have you considered that the problem isn't people not BOTHERING to set the correct settings but not KNOWING that they have to! I've just started using Selenium and have read the docs back to front, many of the wiki pages, the FAQs and yet this is the first I've heard of it. Perhaps if it is such a big deal that "this information is useful to disseminate to a larger number." then larger emphasis should be placed on it in the official guidance? That way less people will ask about it and you can go back to being happy upbeat Selenium Contributor and no more kittens have to be harmed!

    ReplyDelete
    Replies
    1. Scratch the really jaded part please Jim, let's go with 'somewhat jaded' as I think 'really' is a bit of an exaggeration. :)

      Delete
    2. Scratch the 'somewhat jaded' part Jim and let's go with 'perfectly natural developer response to the number of times you've been asked the same question and had to give the same answer.' !!

      Reading up on Protected Mode now and the amount of times it crops up across the Google group and bug tracker is unreal, sure glad I didn't jump the gun and make any wild accusations of jadedness before reading up more on it...

      Delete
    3. I'd just like to point out that the official Selenium documentation on the IE driver[1], there is a note in that portion's info section to check the wiki page[2], and furthermore to pay special attention to the Required Configuration section on that page[3]. Admittedly, this might not be emphasized enough up front, but as you say, it's been a known configuration thing for long enough that web searches should turn up this solution pretty quickly. Glad you were able to find your way here, and get it figured out.

      [1] http://docs.seleniumhq.org/docs/03_webdriver.jsp#internet-explorer-driver
      [2] https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver
      [3] https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver#required-configuration

      Delete
  21. HI,
    My click function is not working for IE browser please suggest the required solution one is to set desired capability.

    ReplyDelete
  22. I'm an automation QA tester, looking for answers
    I cannot seem to pass this error for any reason with internet explorer.. basically the ie windows opens, and that's it.

    I'm using protractor 5.3.0, node 9.6.1

    Failed: JavaScript error
    Build info: version: '3.9.1', revision: '63f7b50', time: '2018-02-07T22:42:28.403Z'
    System info: host: 'DEV-WORKSTATION', ip: '172.31.24.173', os.name: 'Windows Server 2016', os.arch: 'amd64', os.version: '10.0', java.version: '9.0.4'
    Driver info: driver.version: unknown
    JavascriptError: JavaScript error
    Build info: version: '3.9.1', revision: '63f7b50', time: '2018-02-07T22:42:28.403Z'
    System info: host: 'DEV-WORKSTATION', ip: '172.31.24.173', os.name: 'Windows Server 2016', os.arch: 'amd64', os.version: '10.0', java.version: '9.0.4'
    Driver info: driver.version: unknown

    ReplyDelete
  23. No, it's not that easy.
    I work in a corporate setting and someone else sets all of that - all of the settings you're talking about are greyed out for me. I even have local admin credentials and I still can't enable protected mode for a zone.

    ReplyDelete