Support the W3C webdriver spec

[stof]

Support for the JsonWire selenium protocol is fading out in favor of the W3C spec:

• Firefox's geckodriver implements only the W3C protocol
• Safari's driver uses the W3C driver by default (see https://github.com/minkphp/Mink/issues/764)
• Chrome's driver is working on supporting the W3C protocol (they still implement JsonWire for now)
• Edge's driver uses the W3C driver by default in its next release (so same situation than Safari will happen soon: https://docs.microsoft.com/en-us/microsoft-edge/webdriver#webdriver-server-command-line-flags)

We will have some work to do to make this happen.

The instaclick/php-webdriver library that we are using has a beta version of 2.0 which adds support for interacting with a W3C server. But they won't provide an abstraction layer to make the same code work for both variants (see https://github.com/instaclick/php-webdriver/issues/84). It will be up to our code to choose the right way (and we will need https://github.com/instaclick/php-webdriver/issues/86 to be able to know which one we need). Due to the fact that we expose the webdriver session to userland, we would have to consider this upgrade as a BC break.

Another option could be to migrate to facebook/webdriver, which seems a bit more popular this days (https://packagist.org/?query=webdriver). It is currently adding support for the W3C spec (WIP is at https://github.com/facebook/php-webdriver/pull/560), and does so by abstracting the differences internally). Doing such migration would also require making it a major release, as we would change the API we expose when exposing the session.

[stof]

https://github.com/minkphp/MinkSelenium2Driver/issues/288 was created to refer to the first solution for this topic (but does not describe the implication, and does not seem to care about the W3C support in the PR implementing it)

[stof]

#287 should be done first before this work (we should switch to Chrome on Travis, as there is no way to run recent Firefox versions with the JsonWire protocol).

[stof]

The more I look at instaclick/php-webdriver 2 "supporting" the WEC protocol (currently in RC), the more I think we should rather migrate to the facebook package (for which the PR supporting the W3C protocol inside their abstraction is almost ready AFAICT).

[stof]

Btw, if we look at https://packagist.org/packages/instaclick/php-webdriver/dependents and the download counts on https://packagist.org/?query=webdriver, I think most of the downloads of instaclick/php-webdriver are coming from Mink using it (our driver has 11 176 049 downloads, the library has 11 517 849, so this only means 340k downloads not coming from us, which is 3%). Among the 12 packages depending on instaclick/php-webdriver on Packagist, here is my analysis:

• 1 is us
• 2 are (abandoned) forks of us, (one direct fork, and a ghost-driver which duplicated our codebase instead of depending on us)
• 3 are about locking exact versions of deps in the Drupal ecosystem and so are related to us (as Drupal is using Mink)
• 1 is an abandoned experiment from the Laravel ecosystem, but laravel/dusk has been based on the facebook library instead.
• 1 is a fork of the previous experiment
• the remaining 4 look abandoned (last commits either in 2016 or 2014)

So we are the only ones keeping instaclick/php-webdriver alive in the open.

[aik099]

https://github.com/minkphp/MinkSelenium2Driver/issues/293#issuecomment-426670405

Unfortunately Facebook Web Driver is now requiring PHP 7 and that was main reason why PR still isn't merged.

[stof]

@aik099 as we need to make a new major release of the driver anyway, I'm fine with requiring PHP 7.1+ for it. All older PHP versions will be EOLed by the end of December.

People really wanting to use PHP 5.x could still use Selenium2Driver 1.x and use Chrome (or another browser with a driver supporting the legacy protocol)

[aik099]

Agreed. We'll then have new driver require PHP 7.1, but Mink itself stays with current 5.x requirement.

[stof]

Well, Mink itself will probably bump its own requirement in the future too, but I don't have a roadmap for that (as long as we don't make changes benefiting from it, there is no strong need)

[nirbhaypatil]

just wanted to know how to use instaclick/php-webdriver:2.0.0-RC2 with mink and mink extension ? dependencies mismatch error comes up.

{ "require": { "behat/behat":"~3", "behat/mink": "1.7.1", "behat/mink-extension": "2.2", "behat/mink-selenium2-driver": "1.3.1", "behat/mink-goutte-driver": "1.2.1", "drupal/drupal-extension":"3.1.5", "sensiolabs/behat-page-object-extension": "dev-master" }, "config": { "bin-dir": "bin/" } }

Your requirements could not be resolved to an installable set of packages.

[stof]

currently, you cannot do it directly.

[nirbhaypatil]

even with facebook/webdriver + minkextension? can you suggest other ways to do .

[stof]

I don't understand your question. You asked whether you can use instaclick/php-webdriver:2.0.0-RC2, and now you say even with facebook/webdriver ?. I don't see how they are related.

[nirbhaypatil]

Sorry about confusion ,my question is Can we use

1. mink + minkextension + instaclick/php-webdriver:2.0.0-RC2 OR
2. mink + minkextension + facebook/webdriver to execute tests on firefox > 48

[stof]

none of that is possible today, which is precisely what this issue is about:

• using facebook/webdriver would involve rewriting the driver entirely, as it is a different library. You cannot just install it and expect it to be used
• installing instaclick/php-webdriver:2.0.0-RC2 will not make it work with recent Firefox, even if you trick composer into accepting it. instaclick/php-webdriver does not abstract the differences between the W3C and JsonWire protocol, but exposes APIs supporting both and expects the calling code to use the right one, which MinkExtension does not do it yet. And we cannot do it based on their RC2 releases:
  • they don't expose whether the session is a W3C or JsonWire one, so we would have to guess it currently (without having the info to detect it reliably): https://github.com/instaclick/php-webdriver/issues/86
  • their timeout handling (one of the few APIs they actually abstract) is broken with W3C: https://github.com/instaclick/php-webdriver/issues/88
  • the error handling is broken with W3C: https://github.com/instaclick/php-webdriver/issues/87

So we are still discussing in which direction we want to rewrite the driver (supporting the W3C spec on top of instaclick/php-webdriver:2 also involves lots of rewriting in the driver).

[oleg-andreyev]

@stof recently we've been trying to migrate this library to facebook/php-webdriver, mostly because of syn.js which bundled into this library and because instaclick/php-webdriver is not supported that actively.

So far my results: Diff: https://github.com/minkphp/MinkSelenium2Driver/compare/master...oleg-andreyev:master Travis Build (with Chrome): https://travis-ci.org/oleg-andreyev/MinkSelenium2Driver/jobs/462514905 Travis Build (with FF): https://travis-ci.org/oleg-andreyev/MinkSelenium2Driver/jobs/462514904 Travis: https://travis-ci.org/oleg-andreyev/MinkSelenium2Driver/builds/462514900

There are few issues with Chrome eg: Keyboard event, but it it's related to assert of driver-testsuite. Also there are issues with facebook/webdriver and with FF

[phil-davis]

Note: chrome+chromedriver 75 has recently come out and now defaults to w3c So now I think MinkSelenium2Driver users need to set chromeOption w3c to false

Also see https://github.com/SeleniumHQ/docker-selenium/issues/922 where I am having this trouble with the latest https://github.com/SeleniumHQ/docker-selenium 3.141.59-palladium which has chrome 75.

[AlexSkrypnyk]

@phil-davis Looks like we were debugging this at the same time :))

This is the full fix in behat.yml:

  extensions:
    Behat\MinkExtension:
      selenium2:
         capabilities: { "browser": "chrome", "version": "*", "marionette": true, "extra_capabilities": { "chromeOptions": { "w3c": false } } }

Until https://github.com/instaclick/php-webdriver implements support for w3c protocol, we need to use the workaround above.

[phil-davis]

@alexdesignworks I have been trying that in https://github.com/owncloud/core/pull/35445 In BEHAT_PARAMS I end up with

"capabilities": {"marionette":null, "browser": "chrome", "platform": "Linux", "name": "owncloud/core - ", "extra_capabilities": {"chromeOptions":{"w3c":false},"maxDuration":"3600"}}

https://drone.owncloud.com/owncloud/core/18090/681

But I get a WebDriver Exception:

    When user "user1" logs in using the webUI                                                           # WebUILoginContext::theUserLogsInUsingTheWebUI()
      WebDriver\Exception\CurlExec: Webdriver http error: 400, payload :{"value":{"error":"invalid argument","message":"invalid argument: missing command parameters","stacktrace":"#0 0x55c24baff299 \u003Cunknown>\n"}} in /drone/src/vendor-bin/behat/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:155
      Stack trace:
      #0 /drone/src/vendor-bin/behat/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php(132): WebDriver\Exception::factory(-1, 'Webdriver http ...')
      #1 /drone/src/vendor-bin/behat/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php(218): WebDriver\AbstractWebDriver->curl('POST', '/click', NULL)
      #2 /drone/src/vendor-bin/behat/vendor/instaclick/php-webdriver/lib/WebDriver/Container.php(224): WebDriver\AbstractWebDriver->__call('click', Array)
      #3 /drone/src/vendor-bin/behat/vendor/behat/mink-selenium2-driver/src/Selenium2Driver.php(781): WebDriver\Container->__call('click', Array)
      #4 /drone/src/vendor-bin/behat/vendor/behat/mink-selenium2-driver/src/Selenium2Driver.php(769): Behat\Mink\Driver\Selenium2Driver->clickOnElement(Object(WebDriver\Element))
      #5 /drone/src/vendor-bin/behat/vendor/behat/mink/src/Element/NodeElement.php(153): Behat\Mink\Driver\Selenium2Driver->click('(//html/.//*[./...')
      #6 /drone/src/tests/acceptance/features/lib/LoginPage.php(71): Behat\Mink\Element\NodeElement->click()
      #7 /tmp/ProxyManagerGeneratedProxy__PM__PageLoginPageGenerated6312256563dff9947346b9792a817253.php(34): Page\LoginPage->loginAs('user1', '1234', 'FilesPage')

Any clues are welcome!

[mnocon]

@phil-davis We're having the same error with our test suite (after setting w3c:false). I'll keep investigating, but for now we will stay with 3.141.59-oxygen to unblock tests.

[AlexSkrypnyk]

@phil-davis I'm getting the same. That is actually comes from Selenium2Driver.php

    private function clickOnElement(Element $element)
    {
        try {
            // Move the mouse to the element as Selenium does not allow clicking on an element which is outside the viewport
            $this->wdSession->moveto(array('element' => $element->getID()));
        } catch (UnknownCommand $e) {
            // If the Webdriver implementation does not support moveto (which is not part of the W3C WebDriver spec), proceed to the click
        }

        $element->click(); // <------ This calls click endpoint
    }

[UPDATE] After more debugging it appears that PHP code works correctly and POST url looks like http://chrome:4444/wd/hub/session/1ea30ec323d3b5f3b1f666005be0c622/element/0.602436616654985-1/click, but this error still persists. I'm wondering if ChromeDriver removed backwards compatibility for non-w3c mode for some methods (maybe even accidentally).

[alexsegura]

Just faced the same error, like @mnocon I fixed it by rollbacking to Chrome 74.

I'm thinking about switching to Chrome Headless, but I'm not sure it would fix the problem.

[individual-it]

There is a bug in the chrome driver https://bugs.chromium.org/p/chromedriver/issues/detail?id=2943 workarounds in facebook/php-webdriver https://github.com/facebook/php-webdriver/pull/640 and nightwatch https://github.com/nightwatchjs/nightwatch/pull/2122

[phil-davis]

Where should we do a "fix" like https://github.com/facebook/php-webdriver/pull/640 In this MinkSelenium2Driver repo, or in https://github.com/instaclick/php-webdriver or?

[normanlolx]

Temporarily pinning to Chromedriver 74 seems to be the best workaround for now. See https://github.com/acquia/blt/pull/3708/files/9aa7d8cf28025b957f1e628d9c6de6e63a0a5aac#diff-586532737a140c4f8e2079e2c33fc217

[jimarasim]

When setting w3c to false, use "goog:chromeOptions" instead of "chromeOptions" extra_capabilities: "goog:chromeOptions": w3c: false

Hoping this issue is resolved before Chrome decides to no longer support jsonwire at all.

[emodric]

Is it only me, or this started to work on its own in latest versions of Chrome Driver 75 and even in the new v76?

[phil-davis]

I am using selenium/standalone-chrome-debug in drone-docker. latest of that still fails, and I have to specify 3.141.59-oxygen

[emodric]

Here is a build that works with latest chrome and chrome driver that previously failed: https://travis-ci.org/netgen-layouts/layouts-core/jobs/566143335

Previous failing build: https://travis-ci.org/netgen-layouts/layouts-core/jobs/545643705

[shwetaneelsharma]

@alexdesignworks Thanks a ton for this solution https://github.com/minkphp/MinkSelenium2Driver/issues/293#issuecomment-499458269 However, does it work for headless execution too? I am only able to get it working for browser-based execution

[shwetaneelsharma]

I have got this working on Chrome 75 in headless mode by using the following configuration: `

Behat\MinkExtension:  
    base_url: "your_base_url"  
    browser_name: 'chrome'
    goutte: ~
    javascript_session: selenium2
    selenium2:
      wd_host: http://127.0.0.1:4444/wd/hub
      capabilities:
        browser: chrome
        extra_capabilities:
          chromeOptions:
            args: ['--headless', '--disable-gpu']
            w3c: false`