Selenium-Remote-Driver
Selenium-Remote-Driver copied to clipboard
teodesian
The method close() seems to not be working with Selenium Grid
Hello, I was trying to migrate some scripts which I have using Selenium::Chrome with a local chromedriver to a doker instance of Selenium. So I used the example provided in the documentation to start the docker server:
# To start Docker in Swarm mode, you need to run `docker swarm init`
# To deploy the Grid, `docker stack deploy -c docker-compose-v3-swarm.yml grid`
# Stop with `docker stack rm grid`
# Stop swarm mode `docker swarm leave --force`
version: '3.7'
services:
chrome:
image: selenium/node-chrome:4.1.1-20211217
shm_size: 2gb
environment:
- SE_EVENT_BUS_HOST=selenium-hub
- SE_EVENT_BUS_PUBLISH_PORT=4442
- SE_EVENT_BUS_SUBSCRIBE_PORT=4443
- SE_NODE_SESSION_TIMEOUT=30
deploy:
replicas: 1
entrypoint: bash -c 'SE_OPTS="--host $$HOSTNAME" /opt/bin/entry_point.sh'
edge:
image: selenium/node-edge:4.1.1-20211217
shm_size: 2gb
environment:
- SE_EVENT_BUS_HOST=selenium-hub
- SE_EVENT_BUS_PUBLISH_PORT=4442
- SE_EVENT_BUS_SUBSCRIBE_PORT=4443
deploy:
replicas: 1
entrypoint: bash -c 'SE_OPTS="--host $$HOSTNAME" /opt/bin/entry_point.sh'
firefox:
image: selenium/node-firefox:4.1.1-20211217
shm_size: 2gb
environment:
- SE_EVENT_BUS_HOST=selenium-hub
- SE_EVENT_BUS_PUBLISH_PORT=4442
- SE_EVENT_BUS_SUBSCRIBE_PORT=4443
deploy:
replicas: 1
entrypoint: bash -c 'SE_OPTS="--host $$HOSTNAME" /opt/bin/entry_point.sh'
selenium-hub:
image: selenium/hub:4.1.1-20211217
ports:
- "4442:4442"
- "4443:4443"
- "4444:4444"
Then I created a small script to test it:
use strict;
use warnings;
use utf8;
use open ':std', ':encoding(UTF-8)';
use Selenium::Remote::Driver;
use Data::Dumper;
my %settings = (
'browser_name' => 'chrome',
'remote_server_addr' => '127.0.0.1',
'port' => '4444',
'auto_close' => 0,
'pageLoadStrategy' => 'eager',
'extra_capabilities' => {
'goog:chromeOptions' => {
'prefs' => {
'profile' => {
'accept_ssl_certs' => 'true',
}
},
'args' => [
'--ignore-certificate-errors',
'--disable-infobars',
'--enable-automation',
'--start-maximized',
'--disable-dev-shm-usage',
'--no-sandbox',
'--aggressive-cache-discard',
'--disable-cache',
'--disable-application-cache',
'--disk-cache-size=0',
'--disable-gpu',
'--dns-prefetch-disable',
'--no-proxy-server',
'--disable-browser-side-navigation',
],
}
}
);
foreach my $i (0..5){
print("Starting test $i\n");
my $driver = Selenium::Remote::Driver->new(%settings);
$driver->debug_on;
$driver->get('https://google.com');
print($driver->get_title() . "\n");
sleep(5);
$driver->close();
print("Driver closed\n");
}
For some reason it seems to get stucked in the $driver->close() and it will not close the session in the docker container. This is the output of the logs
Starting test 0
Prepping get
Executing get
REQ: POST, http://127.0.0.1:4444/wd/hub/session/9ef0b00727e44c4aa8043cecc7a33e09/url, {"url":"https://google.es"}
RES: {"value":null}
Prepping getTitle
Executing getTitle
REQ: GET, http://127.0.0.1:4444/wd/hub/session/9ef0b00727e44c4aa8043cecc7a33e09/title, {}
RES: {"value":"Google"}
Google
Prepping close
Executing close
REQ: DELETE, http://127.0.0.1:4444/wd/hub/session/9ef0b00727e44c4aa8043cecc7a33e09/window, {}
RES: {
"value": {
"error": "unknown error",
"message": "java.net.ConnectException: Connection refused: localhost\u002f127.0.0.1:36558",
"stacktrace": "java.io.UncheckedIOException: java.net.ConnectException: Connection refused: localhost\u002f127.0.0.1:36558\n\tat org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:80)\n\tat org.openqa.selenium.remote.http.RetryRequest.lambda$apply$6(RetryRequest.java:83)\n\tat net.jodah.failsafe.Functions.lambda$get$0(Functions.java:48)\n\tat net.jodah.failsafe.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:66)\n\tat net.jodah.failsafe.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:66)\n\tat net.jodah.failsafe.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:66)\n\tat net.jodah.failsafe.Execution.executeSync(Execution.java:128)\n\tat net.jodah.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:379)\n\tat net.jodah.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:68)\n\tat org.openqa.selenium.remote.http.RetryRequest.lambda$apply$7(RetryRequest.java:83)\n\tat org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)\n\tat org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)\n\tat org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:51)\n\tat org.openqa.selenium.remote.http.RetryRequest.lambda$apply$6(RetryRequest.java:83)\n\tat net.jodah.failsafe.Functions.lambda$get$0(Functions.java:48)\n\tat net.jodah.failsafe.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:66)\n\tat net.jodah.failsafe.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:66)\n\tat net.jodah.failsafe.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:66)\n\tat net.jodah.failsafe.Execution.executeSync(Execution.java:128)\n\tat net.jodah.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:379)\n\tat net.jodah.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:68)\n\tat org.openqa.selenium.remote.http.RetryRequest.lambda$apply$7(RetryRequest.java:83)\n\tat org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)\n\tat org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)\n\tat org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:110)\n\tat org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)\n\tat org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:92)\n\tat org.openqa.selenium.grid.node.ProtocolConvertingSession.execute(ProtocolConvertingSession.java:75)\n\tat org.openqa.selenium.grid.node.local.SessionSlot.execute(SessionSlot.java:123)\n\tat org.openqa.selenium.grid.node.local.LocalNode.executeWebDriverCommand(LocalNode.java:387)\n\tat org.openqa.selenium.grid.node.ForwardWebDriverCommand.execute(ForwardWebDriverCommand.java:35)\n\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:373)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.node.Node.execute(Node.java:240)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\nCaused by: java.net.ConnectException: Connection refused: localhost\u002f127.0.0.1:36558\n\tat org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179)\n\tat org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108)\n\tat org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28)\n\tat org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)\n\tat
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)\n\tat
io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)\n\tat io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)\n\tat io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\t... 1 more\nCaused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost\u002f127.0.0.1:36558\nCaused by: java.net.ConnectException: Connection refused\n\tat java.base\u002fsun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat java.base\u002fsun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)\n\tat io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\n"
}
}
Error while executing command: unknown error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:36558 at C:/Strawberry/perl/site/lib/Selenium/Remote/Driver.pm line 411.
at C:/Strawberry/perl/site/lib/Selenium/Remote/Driver.pm line 361.
It looks like the timeout I have in the dokcer containers triggers the exception, but otherwise it stays there forever and never moves to the next iteration of the loop.
Tested using version 1.46
looks like it couldn't communicate with the underlying browser driver binary. Your guess is as good as mine regarding why.
Given you are using chrome explicitly in your example, you may want to try a dedicated chromedriver container. https://github.com/RobCherry/docker-chromedriver
I tried something similar in python and there it seems to work with all browsers of the grid:
from selenium import webdriver
import time
def set_options() -> dict:
"""
Sets chrome options for Selenium.
Chrome options for headless browser is enabled.
"""
# options = webdriver.ChromeOptions()
# options = webdriver.FirefoxOptions()
options = webdriver.EdgeOptions()
# options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument('--ignore-certificate-errors')
options.add_argument('--allow-insecure-localhost') # differ on driver version. can ignore.
options.add_argument('--ignore-ssl-errors=yes')
options.add_argument("--allow-running-insecure-content")
options.add_argument('add_experimental_option("excludeSwitches",["ignore-certificate-errors"])')
options.add_argument('--allow-running-insecure-content')
options.add_argument('--ignore-certificate-errors')
# chrome_prefs = {}
# chrome_prefs["profile.default_content_settings"] = {"images": 2}
return options
if __name__ == "__main__":
selenium_grid_url = "http://127.0.0.1:4444/wd/hub"
for i in range(0,5):
# Instantiate an instance of Remote WebDriver with the desired capabilities.
print(f'Creating driver {i}...')
capabilities = set_options().to_capabilities()
capabilities['acceptInsecureCerts'] = True
driver = webdriver.Remote(
command_executor='http://127.0.0.1:4444/wd/hub',
desired_capabilities=capabilities
)
print('Getting site...')
driver.get('https://google.com')
print(driver.title)
print(f'Browser version: {driver.capabilities["browserVersion"]}')
time.sleep(5)
driver.quit()
I changed the close by quit but it still gets stucked in the same way.
I don't really have context on what you are doing, but maybe you need to do something like this?
require LWP::Simple;
my $_url = "http://localhost:$selenium_port/extra/LifecycleServlet?action=shutdown";
my $_content = LWP::Simple::get $_url;
My idea was to move tests which run locally with a chromedriver, geckodriver... into a remote server which provides browsers dynamically as they are needed and close them when the test is finished. But I'm having this issue closing the browser, so if only one is available for chrome for example it will get stucked there.
curious -- do things shutdown cleanly if you omit close() and just let the destructor clean up? If so that might suggest something I can do about this.
Yes It does work in that scenario. With the virtual environment variable - SE_NODE_SESSION_TIMEOUT=30 it does close the session:
use strict;
use warnings;
use utf8;
use open ':std', ':encoding(UTF-8)';
use Selenium::Remote::Driver;
use Data::Dumper;
my %settings = (
'browser_name' => 'chrome',
'remote_server_addr' => '127.0.0.1',
'port' => '4444',
'auto_close' => 0,
'pageLoadStrategy' => 'eager',
'extra_capabilities' => {
'goog:chromeOptions' => {
'prefs' => {
'profile' => {
'accept_ssl_certs' => 'true',
}
},
'args' => [
'--ignore-certificate-errors',
'--disable-infobars',
'--enable-automation',
'--start-maximized',
'--disable-dev-shm-usage',
'--no-sandbox',
'--aggressive-cache-discard',
'--disable-cache',
'--disable-application-cache',
'--disk-cache-size=0',
'--disable-gpu',
'--dns-prefetch-disable',
'--no-proxy-server',
'--disable-browser-side-navigation',
],
}
}
);
foreach my $i (0..5){
print("Starting test $i\n");
my $driver = Selenium::Remote::Driver->new(%settings);
$driver->debug_on;
$driver->get('https://google.es');
print($driver->get_title() . "\n");
# sleep(5);
# $driver->quit();
# print("Driver closed\n");
}
Starting test 0
Prepping get
Executing get
REQ: POST, http://127.0.0.1:4444/wd/hub/session/f81f9841df219aa1b2bf5fc93e552163/url, {"url":"https://google.es"}
RES: {"value":null}
Prepping getTitle
Executing getTitle
REQ: GET, http://127.0.0.1:4444/wd/hub/session/f81f9841df219aa1b2bf5fc93e552163/title, {}
RES: {"value":"Google"}
Google
Starting test 1
Prepping get
Executing get
REQ: POST, http://127.0.0.1:4444/wd/hub/session/717a06614ddfd62143b84b603629887a/url, {"url":"https://google.es"}
RES: {"value":null}
Prepping getTitle
Executing getTitle
REQ: GET, http://127.0.0.1:4444/wd/hub/session/717a06614ddfd62143b84b603629887a/title, {}
RES: {"value":"Google"}
Google
Starting test 2
The problem with this is that you have to wait that time and it may not be deterministic. For example a test which requires some processing may take longer to compute some input for selenium and the browser would be dead when you actually want to use it.
Indeed. But the fact that the destructor doesn't explode gives me a thread to pull on which might be able to fix your issue -- there's probably some extra code in DESTROY() that does things better than close() in your situation.
I won't be able to investigate this further until at least the weekend.
Hello, I have the same issue than you with the quit method from the Driver.pm. Taking a look at the stack trace it seems that it fails when we call the $self->_execute_command($res); (line 884)
To be precise, it fails at this instruction: return $self->commands_v3->parse_response( $res, $resp )if $self->{is_wd3};
Those are the capabilities that I sent to the chrome driver:
my $driverCapabilities = { 'remote_server_addr' => $server, 'browser_name' => chrome, 'extra_capabilities' => { 'acceptInsecureCerts' => \1, 'se:name' => $testName, 'goog:chromeOptions' => { 'args' => ['incognito', 'no-sandbox'] } } };
For the server It was actually pointing to a docker in a aws instance which has the selenium 4.1.2. I also did some test in local raising a standalone server of the latest version of it 4.3.0 but the issue is still there in both cases.
I also tried to call the close method and the DESTROY one but it din't change that much.
By the way with the oldest version of the selenium, (3.149) we were not facing that issue. As we need to migrate to the new selenium grid for our perl test, it might be something that be want to investigate a bit more. Our test are being sent properly and the asserts are not failing.
Do you have any news or idea on this issue ?
Let me know if you need some extra information.
Apologies for not being able to follow up on this issue in a timely manner. It managed to fall through the cracks of my schedule.
I'm currently ill, but will look again next week when well.
OK, I confused myself earlier RE destructors, there is no destructor in this module (I've written too many wrappers around this module).
Looked back at the original report here and this jumped out at me:
Connection refused: localhost/127.0.0.1:36558
As I mentioned at the time, this means that the selenium hub couldn't communicate with the local chromedriver...likely because your server address is 'localhost'. While we can talk to the chromedriver binary locally, the selenium hub can't (as localhost to it is the docker environment). Docker hub-spoke arrangements have never worked in my experience without each component server being specified directly as an IP or hostname.
I'm not sure what level of control you have over your local network topology, but I would assume you could at least figure out a way for the docker container to talk to it's host OS.