Non-deterministic hang in ServletHandler startup for Grid Node

[lukeis]

Originally reported on Google Code with ID 8237

What steps will reproduce the problem?
1. Start a Selenium Hub
2. Start a Selenium Node that will connect to the above Hub
3. (Occasionally) The Node appears to hang between 0-40+ minutes
  - Note the difference in time between the following log statements: (Nearly 3.5 minutes)
        09:02:25.204 INFO - Started HttpContext[/,/]
        09:05:58.688 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler@1c664c4f

What is the expected output? What do you see instead?
The delay should be negligable when starting a Node.

Selenium version: 2.44.0
OS: Ubuntu 14.04
JVM: apt-get install openjdk-7-jre-headless

In particular I see this issue in the Selenium Docker images (https://github.com/SeleniumHQ/docker-selenium).
See issue #14 (https://github.com/SeleniumHQ/docker-selenium/issues/14)

I have also seen this issue reproduced in the docker images by repeatedly running the
test.sh script at https://github.com/SeleniumHQ/docker-selenium/test.sh

Reported by mtscout6 on 2014-12-01 17:20:50

[lukeis]

Reported by barancev on 2014-12-01 18:34:10

• Labels added: Component-Grid

[lukeis]

Here is the debug output before it hangs:

15:13:00.158 DEBUG - Starting HttpContext[/,/]
15:13:00.158 DEBUG - Init classloader from null, sun.misc.Launcher$AppClassLoader@78ea5d87
for HttpContext[/,/]
15:13:00.159 DEBUG - Started org.openqa.selenium.server.ProxyHandler in HttpContext[/,/]
15:13:00.159 INFO - Started HttpContext[/,/]
15:13:00.159 DEBUG - Starting HttpContext[/wd,/wd]
15:13:00.159 DEBUG - Init classloader from null, sun.misc.Launcher$AppClassLoader@78ea5d87
for HttpContext[/wd,/wd]
15:13:00.159 DEBUG - Starting org.openqa.jetty.jetty.servlet.ServletHandler@20d89c2
15:13:00.159 DEBUG - New random session seed

Reported by mtscout6 on 2014-12-02 15:14:05

[lukeis]

Try installing haveged.

It seems that selenium uses blocking random numbers source and hangs when kernel has
no entropy.

Reported by eugene.dounar on 2014-12-12 13:31:03

[lukeis]

Doing some further reading I stumbled on a blog post (http://blog.pivotal.io/cloud-foundry-pivotal/features/challenges-with-randomness-in-multi-tenant-linux-container-platforms)
which really explains this problem in better detail than I can. Also, there is a Stack
Overflow (http://stackoverflow.com/questions/26021181/not-enough-entropy-to-support-dev-random-in-docker-containers-running-in-boot2d)
question that suggesting running with docker run -v /dev/urandom:/dev/random ... which
I have found to work for me. Unfortunately there are trade-offs to this solution as
stated the the blog post I linked earlier.

I don't know enough about the core internals of Selenium, what sounds like the best
solution here? How secure do those sessions need to be? If security isn't too much
of a concern here is /dev/urandom sufficient?

Reported by mtscout6 on 2014-12-12 15:55:51

[lukeis]

Reported by barancev on 2014-12-12 16:14:17

[lukeis]

See /java/server/src/org/openqa/jetty/jetty/servlet/AbstractSessionManager.java line
478. This needs to be rewritten. Using crypto-strong RNG, and then seeding it with
it's own value xored with some non-random variables like free memomory makes no sense.

In my opinion there's absolutely no reason to use SecureRandom. java.util.Random LCG
with 48bit seed is more than enough. 

Reported by eugene.dounar on 2014-12-12 22:15:04

[lukeis]

I would like to add this was a very hard problem to diagnose and get a correct solution
for. 

For myself, installing 'haveged' worked a charm for selenium-server-standalone-2.45.0.jar
on Ubuntu 14.04 x64

Worked fine on OSX Yosemite 10.10.3, no issues or extra setup other than the usual.

Reported by shoutsid on 2015-04-21 20:32:39

[lukeis]

Reported by luke.semerau on 2015-09-17 17:47:30

• Labels added: Restrict-AddIssueComment-Commit