[🐛 Bug]: Browser pods not scaling down

[sugrivg]

What happened?

We installed the Selenium-Grid with auto-scaling on Kubernetes v1.29. The environment is as follows: Kubernetes version: 1.29 ContainerD version: 1.6.31 CNI plugin: Calico Helm: 3.14.4 The browser pods are getting scaled-up properly, however the scaling-down is not working properly.

Command used to start Selenium Grid with Docker (or Kubernetes)

helm upgrade selenium-grid docker-selenium/selenium-grid --version 0.29.1  --set autoscaling.enabled=true --set basicAuth.enabled=false --set autoscaling.scaledOptions.maxReplicaCount=1000 --namespace selenium

Relevant log output

Logs of one of the Chrome pods:

kubectl logs -n selenium selenium-grid-selenium-chrome-node-zxw7t-qtq8k
INFO Included extra file "/etc/supervisor/conf.d/chrome-cleanup.conf" during parsing
INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
INFO RPC interface 'supervisor' initialized
CRIT Server 'unix_http_server' running without any HTTP authentication checking
INFO supervisord started with pid 8
INFO spawned: 'browserleftoverscleanup' with pid 9
INFO spawned: 'xvfb' with pid 10
INFO spawned: 'vnc' with pid 11
INFO spawned: 'novnc' with pid 12
INFO spawned: 'selenium-node' with pid 13
INFO exited: browserleftoverscleanup (exit status 0; not expected)
INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Appending Selenium options: --session-timeout 300
Appending Selenium options: --register-period 60
Appending Selenium options: --register-cycle 5
Appending Selenium options: --heartbeat-period 30
Appending Selenium options: --log-level INFO
Generating Selenium Config
Setting up SE_NODE_HOST...
Tracing is disabled
Selenium Grid Node configuration:
[events]
publish = "tcp://selenium-grid-selenium-hub.selenium:4442"
subscribe = "tcp://selenium-grid-selenium-hub.selenium:4443"

[server]
port = "5555"
[node]
grid-url = "http://selenium-grid-selenium-hub.selenium"
session-timeout = "300"
override-max-sessions = false
detect-drivers = false
drain-after-session-count = 1
max-sessions = 1

[[node.driver-configuration]]
display-name = "chrome"
stereotype = '{"browserName": "chrome", "browserVersion": "123.0", "platformName": "Linux", "goog:chromeOptions": {"binary": "/usr/bin/google-chrome"}}'
max-sessions = 1

Starting Selenium Grid Node...
INFO spawned: 'browserleftoverscleanup' with pid 124
INFO success: xvfb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
INFO success: vnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
INFO success: novnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
INFO exited: browserleftoverscleanup (exit status 0; not expected)
INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
INFO [UnboundZmqEventBus.<init>] - Connecting to tcp://selenium-grid-selenium-hub.selenium:4442 and tcp://selenium-grid-selenium-hub.selenium:4443
INFO [UnboundZmqEventBus.<init>] - Sockets created
INFO [UnboundZmqEventBus.<init>] - Event bus ready
INFO spawned: 'browserleftoverscleanup' with pid 143
INFO exited: browserleftoverscleanup (exit status 0; not expected)
INFO [NodeServer.createHandlers] - Reporting self as: http://xxx.xxx.xxx.xxx:5555
INFO [NodeOptions.getSessionFactories] - Detected 1 available processors
INFO [NodeOptions.report] - Adding chrome for {"browserName": "chrome","browserVersion": "123.0","goog:chromeOptions": {"binary": "\u002fusr\u002fbin\u002fgoogle-chrome"},"platformName": "linux","se:noVncPort": 7900,"se:vncEnabled": true} 1 times
INFO [Node.<init>] - Binding additional locator mechanisms: relative
INFO [NodeServer$1.start] - Starting registration process for Node http://xxx.xxx.xxx.xxx:5555
INFO [NodeServer.execute] - Started Selenium node 4.19.1 (revision abe0ee07dc): http://xxx.xxx.xxx.xxx:5555
INFO [NodeServer$1.lambda$start$1] - Sending registration event...
INFO [NodeServer.lambda$createHandlers$2] - Node has been added
INFO spawned: 'browserleftoverscleanup' with pid 155
INFO exited: browserleftoverscleanup (exit status 0; not expected)
INFO gave up: browserleftoverscleanup entered FATAL state, too many start retries too quickly

Operating System

Oracle Linux Server release 8.9

Docker Selenium version (image tag)

selenium-grid-4.19.1-20240402

Selenium Grid chart version (chart version)

0.29.1

[github-actions[bot]]

@sugrivg, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[VietND96]

Can you re-check this issue in the recent chart version release? (latest is 0.32.0)

[VietND96]

I believe this should be fixed in chart version 0.35.x onwards.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.