zalenium icon indicating copy to clipboard operation
zalenium copied to clipboard

Published 20 hours ago • verified publisher icon zalando

Radom failure when run selenium tests in parallel with error info as 'cannot forward the request unexpected end of stream on Connection`

Open DowneyTung opened this issue 6 years ago • 42 comments

Please make sure that you provide enough information for us to help you with this issue. Thank you!

Zalenium Image Version(s):
3.14.0d Docker Version:
18.06.1-ce If using docker-compose, version:
1.22.0

OS: MacOS High Sierra 10.13.6 16GB RAM

Docker Command to start Zalenium:
I use docker compose to start the service

 version: "3"
services:
    zalenium:
        image: "dosel/zalenium"
        container_name: zalenium
        hostname: zalenium
        tty: true
        volumes:
            - '/tmp/videos:/home/seluser/videos'
            - '/var/run/docker.sock:/var/run/docker.sock'
            - '/usr/local/bin/docker:/usr/local/bin/docker'
        ports:
            - 4444:4444
        command: >
          start --chromeContainers 4
                --firefoxContainers 0
                --maxDockerSeleniumContainers 6
                --maxTestSessions 200
                --screenWidth 1200 --screenHeight 900
                --timeZone "America/New_York"
                --videoRecordingEnabled false
                --sauceLabsEnabled false
                --browserStackEnabled false
                --testingBotEnabled false
                --startTunnel false
        environment:
          - HOST_UID
          - HOST_GID
          - SAUCE_USERNAME
          - SAUCE_ACCESS_KEY
          - BROWSER_STACK_USER
          - BROWSER_STACK_KEY
          - TESTINGBOT_KEY
          - TESTINGBOT_SECRET

Expected Behavior -

The test should run without random error

Actual Behavior -

Now I am getting random test failure. It looks like the request to Selenium does not get through and yield a 500 error, this triggers the failure. I have 4 tests in parallel, and 50 test cases in total. There are about 4-5 tests will fail randomly due to this error. Here is the stacktrace of the error,

2018-10-11 13:01:31,197 root INFO Take screenshot and save as 308fe567.png
2018-10-11 13:01:31,199 selenium.webdriver.remote.remote_connection DEBUG GET http://192.168.240.2:4444/wd/hub/session/af67b690272dd60a3c34f90d2467dc1d/screenshot {"sessionId": "af67b690272dd60a3c34f90d2467dc1d"}
2018-10-11 13:01:31,202 urllib3.connectionpool DEBUG Starting new HTTP connection (1): 192.168.240.2:4444
2018-10-11 13:01:39,841 urllib3.connectionpool DEBUG http://192.168.240.2:4444 "GET /wd/hub/session/af67b690272dd60a3c34f90d2467dc1d/screenshot HTTP/1.1" 500 11436
2018-10-11 13:01:39,842 selenium.webdriver.remote.remote_connection DEBUG Finished Request
2018-10-11 13:01:39,844 root INFO Take screenshot error
2018-10-11 13:01:39,846 selenium.webdriver.remote.remote_connection DEBUG GET http://192.168.240.2:4444/wd/hub/session/af67b690272dd60a3c34f90d2467dc1d/screenshot {"sessionId": "af67b690272dd60a3c34f90d2467dc1d"}
2018-10-11 13:01:39,846 urllib3.connectionpool DEBUG Starting new HTTP connection (1): 192.168.240.2:4444
2018-10-11 13:01:43,108 urllib3.connectionpool DEBUG http://192.168.240.2:4444 "GET /wd/hub/session/af67b690272dd60a3c34f90d2467dc1d/screenshot HTTP/1.1" 500 3261
2018-10-11 13:01:43,109 selenium.webdriver.remote.remote_connection DEBUG Finished Request
2018-10-11 13:01:43,109 root ERROR An error ocurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/golem_framework-0.5.0-py3.6.egg/golem/actions.py", line 351, in verify_element_exist_by_text
    driver.get_screenshot_as_file(img_path)
  File "/usr/local/lib/python3.6/site-packages/selenium-3.14.1-py3.6.egg/selenium/webdriver/remote/webdriver.py", line 1032, in get_screenshot_as_file
    png = self.get_screenshot_as_png()
  File "/usr/local/lib/python3.6/site-packages/selenium-3.14.1-py3.6.egg/selenium/webdriver/remote/webdriver.py", line 1064, in get_screenshot_as_png
    return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
  File "/usr/local/lib/python3.6/site-packages/selenium-3.14.1-py3.6.egg/selenium/webdriver/remote/webdriver.py", line 1074, in get_screenshot_as_base64
    return self.execute(Command.SCREENSHOT)['value']
  File "/usr/local/lib/python3.6/site-packages/selenium-3.14.1-py3.6.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.6/site-packages/selenium-3.14.1-py3.6.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)

selenium.common.exceptions.WebDriverException: Message: cannot forward the request unexpected end of stream on Connection{192.168.240.6:40002, proxy=DIRECT hostAddress=/192.168.240.6:40002 cipherSuite=none protocol=http/1.1}

Stacktrace:
    at org.openqa.grid.web.servlet.handler.RequestHandler.process (RequestHandler.java:143)
    at org.openqa.grid.web.servlet.DriverServlet.process (DriverServlet.java:86)
    at org.openqa.grid.web.servlet.DriverServlet.doGet (DriverServlet.java:64)
    at javax.servlet.http.HttpServlet.service (HttpServlet.java:687)
    at javax.servlet.http.HttpServlet.service (HttpServlet.java:790)
    at org.seleniumhq.jetty9.servlet.ServletHolder.handle (ServletHolder.java:860)
    at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1650)
    at io.prometheus.client.filter.MetricsFilter.doFilter (MetricsFilter.java:170)
    at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1637)
    at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle (ServletHandler.java:533)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:143)
    at org.seleniumhq.jetty9.security.SecurityHandler.handle (SecurityHandler.java:548)
    at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:190)
    at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle (SessionHandler.java:1595)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:188)
    at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle (ContextHandler.java:1253)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:168)
    at org.seleniumhq.jetty9.servlet.ServletHandler.doScope (ServletHandler.java:473)
    at org.seleniumhq.jetty9.server.session.SessionHandler.doScope (SessionHandler.java:1564)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:166)
    at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope (ContextHandler.java:1155)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:141)
    at org.seleniumhq.jetty9.server.handler.StatisticsHandler.handle (StatisticsHandler.java:169)
    at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)
    at org.seleniumhq.jetty9.server.Server.handle (Server.java:530)
    at org.seleniumhq.jetty9.server.HttpChannel.handle (HttpChannel.java:347)
    at org.seleniumhq.jetty9.server.HttpConnection.onFillable (HttpConnection.java:256)
    at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:279)
    at org.seleniumhq.jetty9.io.FillInterest.fillable (FillInterest.java:102)
    at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run (ChannelEndPoint.java:124)
    at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:247)
    at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.produce (EatWhatYouKill.java:140)
    at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:131)
    at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:382)
    at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:708)
    at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:626)
    at java.lang.Thread.run (Thread.java:748)

DowneyTung avatar Oct 11 '18 14:10 DowneyTung

per @diemol 's comment on slack, I also attached the docker logs zalenium here for debugging, I think it might be related to the OKhttp3 issues people reported earlier?

16:02:21.388 [Forwarding 42cb3427-bc4e-4ba8-b3c8-65e2dbeac287 (int. key, remote not contacted yet.) to http://172.19.0.5:40001/wd/hub at 4:02:17 PM] INFO  d.z.e.z.p.DockerSeleniumRemoteProxy - Test session started with internal key 42cb3427-bc4e-4ba8-b3c8-65e2dbeac287 and external key 575083f3e3404de98f2b37ba56e6ee45 assigned to remote 5e4ed07fdba2c891dbfa5512d3ff0019aa9ebf9074ab76a667ab8e74fc402fad.
16:02:21.389 [Forwarding 42cb3427-bc4e-4ba8-b3c8-65e2dbeac287 (int. key, remote not contacted yet.) to http://172.19.0.5:40001/wd/hub at 4:02:17 PM] INFO  d.z.e.z.p.DockerSeleniumRemoteProxy - 5e4ed07fdba2c891dbfa5512d3ff0019aa9ebf9074ab76a667ab8e74fc402fad start-video: Video recording is disabled
16:02:21.532 [Forwarding 84f7e026-0a83-4b7e-ae47-967420b20098 (int. key, remote not contacted yet.) to http://172.19.0.6:40002/wd/hub at 4:02:17 PM] INFO  d.z.e.z.p.DockerSeleniumRemoteProxy - Test session started with internal key 84f7e026-0a83-4b7e-ae47-967420b20098 and external key 6a06646d28a166429772a19a304ae668 assigned to remote fc01c2e9939d6885733a7ac1018f8ee4b508ca28bd96c9b6850440377302112e.
16:02:21.541 [Forwarding 84f7e026-0a83-4b7e-ae47-967420b20098 (int. key, remote not contacted yet.) to http://172.19.0.6:40002/wd/hub at 4:02:17 PM] INFO  d.z.e.z.p.DockerSeleniumRemoteProxy - fc01c2e9939d6885733a7ac1018f8ee4b508ca28bd96c9b6850440377302112e start-video: Video recording is disabled
16:02:21.711 [Forwarding 17b9ace7-60cc-4b79-96d6-f3994464cfc7 (int. key, remote not contacted yet.) to http://172.19.0.4:40000/wd/hub at 4:02:17 PM] INFO  d.z.e.z.p.DockerSeleniumRemoteProxy - Test session started with internal key 17b9ace7-60cc-4b79-96d6-f3994464cfc7 and external key fa91022b1d09fca6b18dc21400f30628 assigned to remote 04547ccac8009940d0a997c5e035da5e9917c20bc6e994c8439c2cf1aed1eb88.
16:02:21.711 [Forwarding 17b9ace7-60cc-4b79-96d6-f3994464cfc7 (int. key, remote not contacted yet.) to http://172.19.0.4:40000/wd/hub at 4:02:17 PM] INFO  d.z.e.z.p.DockerSeleniumRemoteProxy - 04547ccac8009940d0a997c5e035da5e9917c20bc6e994c8439c2cf1aed1eb88 start-video: Video recording is disabled
16:02:21.837 [Forwarding 25c63c75-d0d4-440f-b11c-150e38ce3ef6 (int. key, remote not contacted yet.) to http://172.19.0.7:40003/wd/hub at 4:02:17 PM] INFO  d.z.e.z.p.DockerSeleniumRemoteProxy - Test session started with internal key 25c63c75-d0d4-440f-b11c-150e38ce3ef6 and external key f10eb798fc978eaa6518f31da98e2fca assigned to remote 23aedd6315e3c0b7d94162103cf89c4aadc403d9a6031545d3481c23e11aa1f2.
16:02:21.840 [Forwarding 25c63c75-d0d4-440f-b11c-150e38ce3ef6 (int. key, remote not contacted yet.) to http://172.19.0.7:40003/wd/hub at 4:02:17 PM] INFO  d.z.e.z.p.DockerSeleniumRemoteProxy - 23aedd6315e3c0b7d94162103cf89c4aadc403d9a6031545d3481c23e11aa1f2 start-video: Video recording is disabled
16:02:47.912 [qtp1404565079-14] ERROR o.o.g.w.s.handler.RequestHandler - cannot forward the request unexpected end of stream on Connection{172.19.0.7:40003, proxy=DIRECT hostAddress=/172.19.0.7:40003 cipherSuite=none protocol=http/1.1}
java.io.IOException: unexpected end of stream on Connection{172.19.0.7:40003, proxy=DIRECT hostAddress=/172.19.0.7:40003 cipherSuite=none protocol=http/1.1}
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:208)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
	at okhttp3.RealCall.execute(RealCall.java:77)
	at org.openqa.selenium.remote.internal.OkHttpClient.execute(OkHttpClient.java:105)
	at org.openqa.grid.internal.TestSession.sendRequestToNode(TestSession.java:430)
	at org.openqa.grid.internal.TestSession.forward(TestSession.java:237)
	at org.openqa.grid.web.servlet.handler.RequestHandler.forwardRequest(RequestHandler.java:99)
	at org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:133)
	at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:85)
	at org.openqa.grid.web.servlet.DriverServlet.doGet(DriverServlet.java:63)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.seleniumhq.jetty9.servlet.ServletHolder.handle(ServletHolder.java:860)
	at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
	at io.prometheus.client.filter.MetricsFilter.doFilter(MetricsFilter.java:170)
	at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.seleniumhq.jetty9.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
	at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.seleniumhq.jetty9.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.seleniumhq.jetty9.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.seleniumhq.jetty9.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
	at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.seleniumhq.jetty9.server.Server.handle(Server.java:530)
	at org.seleniumhq.jetty9.server.HttpChannel.handle(HttpChannel.java:347)
	at org.seleniumhq.jetty9.server.HttpConnection.onFillable(HttpConnection.java:256)
	at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.seleniumhq.jetty9.io.FillInterest.fillable(FillInterest.java:102)
	at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
	at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
	at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
	at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
	at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: \n not found: limit=0 content=…
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:237)
	at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
	... 57 common frames omitted
16:02:47.943 [Terminate Test Session int id: [25c63c75-d0d4-440f-b11c-150e38ce3ef6] ext id: [f10eb798fc978eaa6518f31da98e2fca] container: [23aedd6315e3c0b7d94162103cf89c4aadc403d9a6031545d3481c23e11aa1f2]] INFO  d.z.e.z.p.DockerSeleniumRemoteProxy - 23aedd6315e3c0b7d94162103cf89c4aadc403d9a6031545d3481c23e11aa1f2 AFTER_SESSION command received. Cleaning up node for reuse, used 1 of max 200
16:02:48.362 [Terminate Test Session int id: [25c63c75-d0d4-440f-b11c-150e38ce3ef6] ext id: [f10eb798fc978eaa6518f31da98e2fca] container: [23aedd6315e3c0b7d94162103cf89c4aadc403d9a6031545d3481c23e11aa1f2]] INFO  d.z.e.z.c.DockerContainerClient - http://172.19.0.7:40003 [bash, -c, notify 'Zalenium', 'TEST COMPLETED', --icon=/home/seluser/images/completed.png]

DowneyTung avatar Oct 11 '18 20:10 DowneyTung

I attched the whole logs here docker-zalenium.log Also, I attache my github repo that with the code that may be helpful to reproduce the issue. https://github.com/DowneyTung/golem/tree/H5G

DowneyTung avatar Oct 11 '18 20:10 DowneyTung

HI @diemol we are also getting same error

zalenium version:- from docker pulled latest image

os :- linux

docker command to start : sudo docker run --rm -ti --name zalenium:3.10.0b -p 4444:4444 -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/videos:/home/seluser/videos -e WAIT_FOR_AVAILABLE_NODES=true -e SAUCE_USERNAME -e SAUCE_ACCESS_KEY -e SAUCE_LABS_URL -e "zalenium_https_proxy=http://proxy.ops.xxxx-xxxx.org:8080" -e "zalenium_http_proxy=http://proxy.ops.xx-xxxx.org:8080" -e "zalenium_no_proxy=localhost,127.0.0.1,cloud.xxx-xxxx.org,172.17.0.{1-255}"-e ZALENIUM_EXTRA_JVM_PARAMS='-Dwebdriver.http.factory=apache -Dhttps.proxyHost=proxy.ops.xxxx-xxx.org -Dhttps.proxyPort=8080' --privileged dosel/zalenium start --sauceLabsEnabled false --maxTestSessions 1000 --desiredContainers 20 --maxDockerSeleniumContainers 20

attached logs zalenium logs.txt

kunal-piplani avatar Oct 28 '18 06:10 kunal-piplani

@kunal-piplani your log does not look like the log provided initially for this issue. Please open a new issue with in a concise way to reproduce the issue.

diemol avatar Oct 29 '18 14:10 diemol

@DowneyTung when it fails, does it always fail in the same action? Clicking, uploading a file, etc... I see in the stacktrace that it fails while taking a screenshot, but I wonder if that is always the case.

diemol avatar Oct 29 '18 19:10 diemol

@diemol it is not happening for the same action. Sometimes when i wait for a attribute to display...so I think it happened randomly.

DowneyTung avatar Oct 30 '18 18:10 DowneyTung

Hi team!

With that bulk of testing you should really look at your ecosystem. Running a heap on a container will undoubtedly give you some random occurrences because the vm simply cant cope with the strain.

What we should be looking at is your machine performance whilst these parallel runs are occurring. Zalenium is a great tool, but as the developers themselves state, this is a throw away grid to be run locally and they suggest alternative cloud testing solutions like BrowserStack/Saucelabs.

Multi-threading and running tests in parallel can get tricky. It is highly recommended that one restarts/refreshes the hub as well as the node regularly. This is to ensure any memory leaks are taken care off among other things like browser temp files etc.

I hope this helps you. Cheers SP

suveerprithipal avatar Oct 30 '18 20:10 suveerprithipal

Hi @DowneyTung , if you are using TestNG, then try with version 6.14.3. I saw a similar problem when my tests were running with TestNG version 6.11. And it worked without using the ZALENIUM_EXTRA_JVM_PARAMS with TestNG 6.14.3.

I actually found this below config to be the most stable with my framework. I have over 3000 tests.

Selenium server: 3.12.0 Docker version 18.06.1-ce, build e68fc7a elgalu/selenium:3.12.0-p10 dosel/zalenium:3.12.0g

Docker command: docker run --rm -d --name zalenium -p 4444:4444 --log-driver=none -v /var/run/docker.sock:/var/run/docker.sock -v --privileged dosel/zalenium:3.12.0g start --desiredContainers 15 --maxTestSessions 99 --videoRecordingEnabled false --maxDockerSeleniumContainers 16 --seleniumImageName elgalu/selenium:3.12.0-p10

mfuzalam avatar Nov 02 '18 16:11 mfuzalam

@mza031 I am not using TestNG.... I am using python...but will definitely try the version you are using...

DowneyTung avatar Nov 02 '18 17:11 DowneyTung

I have been getting a similar issue (also intermittently, and not occurring on a specific action), it no longer occurred when rolling back to 3.14.0g but began occurring as soon as I rolled forward to 3.141.59a

jordan-mace avatar Dec 07 '18 03:12 jordan-mace

Hi all, We need your help, we know that some people are bumping into this issue, so we would be grateful if someone provided a test suite that we could use to reproduce the issue.

It could be something where the instructions are "you need to run this at least 100 times to get the error" or something like that, it does not need to be an exact test case that reproduces it 100% of the times. Ideally it should be a GitHub repo link where we can just run a command to run the tests.

What we know so far:

  • ZALENIUM_EXTRA_JVM_PARAMS=-Dwebdriver.http.factory=apache does not work anymore because since Selenium 3.141.0, the Apache Http Client was removed from Selenium itself.
  • The issue seems to be this one, but we haven't figured out under what conditions it happens, and what to do when it happens.

diemol avatar Dec 19 '18 14:12 diemol

@DowneyTung I tried to use the repo you linked above, but it seems the tests are not working anymore, could you update them please? (Cool framework by the way!)

diemol avatar Dec 19 '18 14:12 diemol

@diemol Sure. I haven't update the code for a while as I moved the repo to private due to company privacy policy. But I will update the code to make it workable again for the public users. Will let you know once I am finished. Thanks

DowneyTung avatar Dec 20 '18 04:12 DowneyTung

@diemol I updated the test code and you can check it out and see whether it works for you. Let me know if you have any questions https://github.com/DowneyTung/golem/tree/H5G

DowneyTung avatar Dec 20 '18 19:12 DowneyTung

@diemol Am facing the same issue,am running a protractor test in zalenium and my test will be taking close to 180 seconds to complete,but the container get closed with error message as below around 120 seconds,Kindly let me know the workaround for this issue.

WebDriverError: cannot forward the request Connect to 172.17.0.14:40009 [/172.17.0.14] failed: Connection refused (Connection refused)[39m

Balajiiguru avatar Jan 19 '19 17:01 Balajiiguru

Hi all,

I got the same error, when I run a test with more than 6 minutes.

Log of the zalenium:

05:39:06.839 [qtp1591916281-1203] ERROR o.o.g.w.s.handler.RequestHandler - cannot forward the request unexpected end of stream on Connection{172.19.0.4:40001, proxy=DIRECT hostAddress=/172.19.0.4:40001 cipherSuite=none protocol=http/1.1}
java.io.IOException: unexpected end of stream on Connection{172.19.0.4:40001, proxy=DIRECT hostAddress=/172.19.0.4:40001 cipherSuite=none protocol=http/1.1}
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:208)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at org.openqa.selenium.remote.internal.OkHttpClient$Factory$1.lambda$createClient$1(OkHttpClient.java:152)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
	at okhttp3.RealCall.execute(RealCall.java:77)
	at org.openqa.selenium.remote.internal.OkHttpClient.execute(OkHttpClient.java:103)
	at org.openqa.grid.internal.TestSession.sendRequestToNode(TestSession.java:422)
	at org.openqa.grid.internal.TestSession.forward(TestSession.java:229)
	at org.openqa.grid.web.servlet.handler.RequestHandler.forwardRequest(RequestHandler.java:99)
	at org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:133)
	at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:85)
	at org.openqa.grid.web.servlet.DriverServlet.doPost(DriverServlet.java:69)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.seleniumhq.jetty9.servlet.ServletHolder.handle(ServletHolder.java:865)
	at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
	at io.prometheus.client.filter.MetricsFilter.doFilter(MetricsFilter.java:170)
	at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.seleniumhq.jetty9.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.seleniumhq.jetty9.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.seleniumhq.jetty9.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
	at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.seleniumhq.jetty9.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
	at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.seleniumhq.jetty9.server.Server.handle(Server.java:503)
	at org.seleniumhq.jetty9.server.HttpChannel.handle(HttpChannel.java:364)
	at org.seleniumhq.jetty9.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.seleniumhq.jetty9.io.FillInterest.fillable(FillInterest.java:103)
	at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: \n not found: limit=0 content=…
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:237)
	at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
	... 61 common frames omitted

Log of the elaglu/docker-selenium:

2019-01-31 05:38:05,241 INFO spawned: 'video-rec' with pid 15663
2019-01-31 05:38:05,241 INFO success: video-rec entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2019-01-31 05:39:07,549 INFO waiting for video-rec to stop
2019-01-31 05:39:09,549 INFO waiting for video-rec to stop
2019-01-31 05:39:11,554 INFO waiting for video-rec to stop
2019-01-31 05:39:13,391 INFO stopped: video-rec (exit status 0)
2019-01-31 05:39:30,727 INFO spawned: 'video-rec' with pid 16193
2019-01-31 05:39:30,728 INFO success: video-rec entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)

Thanks! Let me know if you have any questions.

WarleyGabriel avatar Jan 31 '19 12:01 WarleyGabriel

Hi,

Did anyone managed to find a solution for this issue?

Appreciate your help!

Thanks, Arun

mcafeerun avatar Mar 05 '19 01:03 mcafeerun

Hi,

We're having the same issue, attaching docker log: 02:31:25.985 [qtp79290250-371] ERROR o.o.g.w.s.handler.RequestHandler - cannot forward the request unexpected end of stream on Connection{172.17.0.4:40022, proxy=DIRECT hostAddress=/172.17.0.4:40022 cipherSuite=none protocol=http/1.1} java.io.IOException: unexpected end of stream on Connection{172.17.0.4:40022, proxy=DIRECT hostAddress=/172.17.0.4:40022 cipherSuite=none protocol=http/1.1} at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:208) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at org.openqa.selenium.remote.internal.OkHttpClient$Factory$1.lambda$createClient$1(OkHttpClient.java:152) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) at okhttp3.RealCall.execute(RealCall.java:77) at org.openqa.selenium.remote.internal.OkHttpClient.execute(OkHttpClient.java:103) at org.openqa.grid.internal.TestSession.sendRequestToNode(TestSession.java:422) at org.openqa.grid.internal.TestSession.forward(TestSession.java:229) at org.openqa.grid.web.servlet.handler.RequestHandler.forwardRequest(RequestHandler.java:99) at org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:133) at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:85) at org.openqa.grid.web.servlet.DriverServlet.doPost(DriverServlet.java:69) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.seleniumhq.jetty9.servlet.ServletHolder.handle(ServletHolder.java:865) at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at io.prometheus.client.filter.MetricsFilter.doFilter(MetricsFilter.java:170) at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.seleniumhq.jetty9.security.SecurityHandler.handle(SecurityHandler.java:548) at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle(ContextHandler.java:1340) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.seleniumhq.jetty9.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.seleniumhq.jetty9.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope(ContextHandler.java:1242) at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.seleniumhq.jetty9.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.seleniumhq.jetty9.server.Server.handle(Server.java:503) at org.seleniumhq.jetty9.server.HttpChannel.handle(HttpChannel.java:364) at org.seleniumhq.jetty9.server.HttpConnection.onFillable(HttpConnection.java:260) at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.seleniumhq.jetty9.io.FillInterest.fillable(FillInterest.java:103) at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.EOFException: \n not found: limit=0 content=… at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:237) at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ... 61 common frames omitted

tomerzel87 avatar Mar 20 '19 08:03 tomerzel87

I am having the same issue as well. Is there anything I can do to help?

ablok avatar Apr 30 '19 11:04 ablok

Yes, me as well. Running Docker for Windows. Firewall disabled. docker run --restart unless-stopped -ti --name zalenium -p 4445:4444 -e ZALENIUM_EXTRA_JVM_PARAMS="-Dwebdriver.http.factory=apache" -v /var/run/docker.sock:/var/run/docker.sock -v /e/ZaleniumVideos:/home/seluser/videos -v /e/ZaleniumShare:/tmp/node/home/seluser/Downloads --privileged dosel/zalenium start --screenWidth 1920 --screenHeight 1080 --timeZone "Europe/Berlin" --desiredContainers 6 --videoRecordingEnabled true --keepOnlyFailedTests true --maxDockerSeleniumContainers 10 --maxTestSessions 6

SomeCoolName444 avatar Apr 30 '19 11:04 SomeCoolName444

For me disabling the firewall on the linux server resolved the issue.

Spoke too soon seems like the issues weren't resolved.

tomerzel87 avatar Apr 30 '19 11:04 tomerzel87

I am also getting this error in almost every test run, what can I do to help resolving this issue? What information you need that might help?

ls-sergii-buchovskyi-zz avatar Oct 15 '19 13:10 ls-sergii-buchovskyi-zz

Increasing the resources allocated to the browser pods helped quite a bit.

  cpuRequest: 800m
  cpuLimit: 4000m
  memRequest: 700Mi
  memLimit: 5Gi

zackb avatar Oct 15 '19 16:10 zackb

Thanks for the suggestion to increase resources allocated to pods running browsers. I'll give it a try, too. Since some time already we're also facing trouble in our build pipeline with executing tests with Zalenium. The problem is the same as this issue here.

What kind of information you'd need to debug this issue?

tpiekarski avatar Oct 16 '19 12:10 tpiekarski

I'd recommend to hook Prometheus or a similar tool to monitor performance, then you can correlate failures with performance metrics and find out how much more or less you need to increase.

diemol avatar Oct 16 '19 12:10 diemol

We are experiencing this as well. Generally we are running 6-10 containers and it seems to happen most often when running a test that has to do some kind of polling on a UI screen.

The strange thing is we've had some instances work for a few hours (obviously other issues there!) and other times, like the past 2 days, it will fail after only a few minutes. Today made it 4 minute and 33 seconds.

In our case, it occurred when we executed some arbitrary javascript due to a click event not being possible. It's difficult to tell if the click was blocked by this same issue, or something related to the page (need to tweak our error logging).

Including trace because one method call is different, and a few line numbers are different, in case it matters:

Original error message: Message: Message: cannot forward the request unexpected end of stream on Connection{172.17.0.7:40189, proxy=DIRECT hostAddress=/172.17.0.7:40189 cipherSuite=none protocol=http/1.1} Stacktrace: at org.openqa.grid.web.servlet.handler.RequestHandler.process (RequestHandler.java:143) at org.openqa.grid.web.servlet.DriverServlet.process (DriverServlet.java:85) at org.openqa.grid.web.servlet.DriverServlet.doPost (DriverServlet.java:69) at javax.servlet.http.HttpServlet.service (HttpServlet.java:707) at javax.servlet.http.HttpServlet.service (HttpServlet.java:790) at org.seleniumhq.jetty9.servlet.ServletHolder.handle (ServletHolder.java:865) at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1655) at io.prometheus.client.filter.MetricsFilter.doFilter (MetricsFilter.java:170) at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1642) at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle (ServletHandler.java:533) at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:146) at org.seleniumhq.jetty9.security.SecurityHandler.handle (SecurityHandler.java:548) at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:257) at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle (SessionHandler.java:1595) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:255) at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle (ContextHandler.java:1340) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:203) at org.seleniumhq.jetty9.servlet.ServletHandler.doScope (ServletHandler.java:473) at org.seleniumhq.jetty9.server.session.SessionHandler.doScope (SessionHandler.java:1564) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:201) at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope (ContextHandler.java:1242) at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:144) at org.seleniumhq.jetty9.server.handler.StatisticsHandler.handle (StatisticsHandler.java:174) at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132) at org.seleniumhq.jetty9.server.Server.handle (Server.java:503) at org.seleniumhq.jetty9.server.HttpChannel.handle (HttpChannel.java:364) at org.seleniumhq.jetty9.server.HttpConnection.onFillable (HttpConnection.java:260) at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:305) at org.seleniumhq.jetty9.io.FillInterest.fillable (FillInterest.java:103) at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.runTask (EatWhatYouKill.java:333) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:310) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.tryProduce (EatWhatYouKill.java:168) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:126) at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:366) at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:765) at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:683) at java.lang.Thread.run (Thread.java:748)

espguitarist33 avatar Oct 18 '19 13:10 espguitarist33

same problem here, causing us some headaches, different tests every time, different actions, we are trying on completely fresh containers every time and still it is happening

selenium.common.exceptions.WebDriverException: Message: cannot forward the request unexpected end of stream on Connection{172.17.0.4:40065, proxy=DIRECT hostAddress=/172.17.0.4:40065 cipherSuite=none protocol=http/1.1}
Stacktrace:
    at org.openqa.grid.web.servlet.handler.RequestHandler.process (RequestHandler.java:143)
    at org.openqa.grid.web.servlet.DriverServlet.process (DriverServlet.java:85)
    at org.openqa.grid.web.servlet.DriverServlet.doGet (DriverServlet.java:63)
    at javax.servlet.http.HttpServlet.service (HttpServlet.java:687)
    at javax.servlet.http.HttpServlet.service (HttpServlet.java:790)
    at org.seleniumhq.jetty9.servlet.ServletHolder.handle (ServletHolder.java:865)
    at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1655)
    at io.prometheus.client.filter.MetricsFilter.doFilter (MetricsFilter.java:170)
    at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1642)
    at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle (ServletHandler.java:533)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:146)
    at org.seleniumhq.jetty9.security.SecurityHandler.handle (SecurityHandler.java:548)
    at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:257)
    at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle (SessionHandler.java:1595)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:255)
    at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle (ContextHandler.java:1340)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:203)
    at org.seleniumhq.jetty9.servlet.ServletHandler.doScope (ServletHandler.java:473)
    at org.seleniumhq.jetty9.server.session.SessionHandler.doScope (SessionHandler.java:1564)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:201)
    at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope (ContextHandler.java:1242)
    at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:144)
    at org.seleniumhq.jetty9.server.handler.StatisticsHandler.handle (StatisticsHandler.java:174)
    at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)
    at org.seleniumhq.jetty9.server.Server.handle (Server.java:503)
    at org.seleniumhq.jetty9.server.HttpChannel.handle (HttpChannel.java:364)
    at org.seleniumhq.jetty9.server.HttpConnection.onFillable (HttpConnection.java:260)
    at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:305)
    at org.seleniumhq.jetty9.io.FillInterest.fillable (FillInterest.java:103)
    at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
    at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.runTask (EatWhatYouKill.java:333)
    at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:310)
    at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.tryProduce (EatWhatYouKill.java:168)
    at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:126)
    at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:366)
    at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:765)
    at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:683)
    at java.lang.Thread.run (Thread.java:748)

symonk avatar Oct 21 '19 08:10 symonk

Same here: 12:04:42.446 [AutoStartProxyPoolPoller] WARN d.z.e.z.c.DockerContainerClient - Failed to fetch container status [ContainerCreationStatus [isCreated=true, c ontainerName=zalenium_o5dheA, containerId=52725e8eadc68bb040dbe6a769197f4b0e610001d830412056d60b37550f6bc6, nodePort=40040]]. com.spotify.docker.client.exceptions.DockerRequestException: Request error: GET unix://localhost:80/containers/52725e8eadc68bb040dbe6a769197f4b0e610001d8304 12056d60b37550f6bc6/json: 200 at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2835) at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2707) at com.spotify.docker.client.DefaultDockerClient.inspectContainer(DefaultDockerClient.java:1085) at de.zalando.ep.zalenium.container.DockerContainerClient.isTerminated(DockerContainerClient.java:659) at de.zalando.ep.zalenium.proxy.DockeredSeleniumStarter.containerHasFinished(DockeredSeleniumStarter.java:350) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) at java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3527) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at de.zalando.ep.zalenium.proxy.AutoStartProxySet.checkContainers(AutoStartProxySet.java:353) at de.zalando.ep.zalenium.proxy.AutoStartProxySet.lambda$0(AutoStartProxySet.java:124) at java.lang.Thread.run(Thread.java:748) Caused by: javax.ws.rs.client.ResponseProcessingException: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input in VALUE_STRING at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 15953] at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:806) at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92) at org.glassfish.jersey.client.JerseyInvocation$5.completed(JerseyInvocation.java:773) at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:198) at org.glassfish.jersey.client.ClientRuntime.access$300(ClientRuntime.java:79) at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:180) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340) at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 common frames omitted Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input in VALUE_STRING at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 15953] at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:618) at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:595) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed(UTF8StreamJsonParser.java:2348) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2357) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getTextCharacters(UTF8StreamJsonParser.java:421) at com.fasterxml.jackson.databind.util.TokenBuffer.copyCurrentEvent(TokenBuffer.java:1022) at com.fasterxml.jackson.databind.util.TokenBuffer.copyCurrentStructure(TokenBuffer.java:1112) at com.fasterxml.jackson.databind.util.TokenBuffer.copyCurrentStructure(TokenBuffer.java:1107) at com.fasterxml.jackson.databind.util.TokenBuffer.copyCurrentStructure(TokenBuffer.java:1107) at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:482) at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1287) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159) at com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:1574) at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:965) at com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:815) at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:256) at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:235) at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:155) at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1085) at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:874) at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:808) at org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:326) at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:803) ... 17 common frames omitted

alpag avatar Oct 22 '19 10:10 alpag

is it possible to use something other than okHttp at present? e.g apache http factory? Also could this be caused by some version of java running on the jenkins slave(s)?

symonk avatar Oct 23 '19 08:10 symonk

also note: this is not caused entirely by 'parallel' runs because I tested it with a sequential single process/thread test run, it still occurs there. We can also erradicate that it is a single command exec causing it, as mine varies on every failure.

its just a random command going through self._exec(command) that dies once every so often.

@diemol is this a selenium issue? I'm not so sure its a 'zalenium' issue?

symonk avatar Oct 23 '19 08:10 symonk