[🐛 Bug]: Selenium process happened memory leak

[kongdengyuan]

What happened?

When I deployed the latest selenium docker (30 nodes) in our VM, I found that the avaliable memory of VM become more and more less due to selenium java process used memory not release.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 1200 8649 0.3 1.5 7850056 500396 ? Sl 03:05 0:59 java -jar /opt/selenium/selenium-server.jar node --bind-host false --config /opt/selenium/config.toml

The VSZ is big and RSS will be continue increase about 900-1400M even if no any task call selenium process. I deployed about 30 node chrome docker on single VM (16 core 32G) , but the selenium docker will use about 18G memory even if no any cases to run.

How can we reproduce the issue?

You can deploy the latest selenium docker , take look (java  -jar /opt/selenium/selenium-server.jar node --bind-host false --config /opt/selenium/config.toml) this process use memory status.

chrome: https://github.com/SeleniumHQ/docker-selenium/releases/tag/4.3.0-20220726
Release 4.3.0-20220726 · SeleniumHQ/docker-selenium

Relevant log output

CONTAINER ID   NAME                CPU %     MEM USAGE / LIMIT    MEM %     NET I/O          BLOCK I/O         PIDS
af5aaca85db3   node-chrome-30-29   0.35%     398.7MiB / 31.4GiB   1.24%     904kB / 164kB    20.5kB / 32.8kB   96
82d8e18d52f9   node-chrome-30-28   0.35%     1.124GiB / 31.4GiB   3.58%     11.2GB / 743MB   7.99MB / 17.1GB   140
7547ec76f5b2   node-chrome-30-27   0.44%     1.05GiB / 31.4GiB    3.35%     13.6GB / 782MB   2.56MB / 20.1GB   140
59179381cc16   node-chrome-30-26   0.32%     1.136GiB / 31.4GiB   3.62%     14.6GB / 767MB   520kB / 22.4GB    137
a6b681e6d981   node-chrome-30-25   0.36%     1.031GiB / 31.4GiB   3.28%     13GB / 753MB     12.4MB / 19.8GB   140
161d03c67c05   node-chrome-30-24   0.31%     1.064GiB / 31.4GiB   3.39%     12.8GB / 764MB   5.84MB / 19.1GB   141
de8b39817432   node-chrome-30-23   0.34%     1.257GiB / 31.4GiB   4.00%     13.8GB / 747MB   79.3MB / 21.1GB   140
e9b22cb1694d   node-chrome-30-22   0.34%     1.127GiB / 31.4GiB   3.59%     12.4GB / 703MB   791kB / 18.6GB    140
82b96fd15a0b   node-chrome-30-21   0.34%     1.04GiB / 31.4GiB    3.31%     12GB / 694MB     5MB / 18.3GB      137
aa8ebc68446b   node-chrome-30-20   0.35%     1.133GiB / 31.4GiB   3.61%     13.4GB / 790MB   1.4MB / 20.7GB    137
5b8e5d4308bd   node-chrome-30-19   0.34%     1.099GiB / 31.4GiB   3.50%     12.4GB / 723MB   602kB / 19GB      141
0f253a735ef1   node-chrome-30-18   0.34%     1.068GiB / 31.4GiB   3.40%     13GB / 798MB     401kB / 19.9GB    140
69f026622d7b   node-chrome-30-17   0.35%     1.131GiB / 31.4GiB   3.60%     13GB / 777MB     655kB / 20.1GB    140
f7761300caa6   node-chrome-30-16   0.36%     1.25GiB / 31.4GiB    3.98%     14.2GB / 805MB   8.52MB / 21.3GB   140
c14581ca6f26   node-chrome-30-15   0.29%     1.017GiB / 31.4GiB   3.24%     13.4GB / 772MB   750kB / 20.7GB    140
8113cf3fa96d   node-chrome-30-14   0.32%     1.046GiB / 31.4GiB   3.33%     13.1GB / 726MB   2.73MB / 20GB     137
7f67be04fed9   node-chrome-30-13   0.47%     1023MiB / 31.4GiB    3.18%     11.6GB / 677MB   754kB / 17.6GB    139
1333d5c8ff69   node-chrome-30-12   0.36%     1.098GiB / 31.4GiB   3.50%     14.1GB / 762MB   401kB / 21.6GB    140
188412fb2de3   node-chrome-30-11   0.31%     404.5MiB / 31.4GiB   1.26%     924kB / 169kB    197kB / 32.8kB    97
49f6e9178e2a   node-chrome-30-10   0.49%     1.085GiB / 31.4GiB   3.46%     11.7GB / 716MB   3.42MB / 17.9GB   139

Operating System

Ubuntu 18.04

Selenium version

selenium-node-chrome:103.0-chromedriver-103.0-grid-4.3.0-20220628

What are the browser(s) and version(s) where you see this issue?

selenium-node-chrome:103.0-chromedriver-103.0-grid-4.3.0-20220628

What are the browser driver(s) and version(s) where you see this issue?

selenium-node-chrome:103.0-chromedriver-103.0-grid-4.3.0-20220628

Are you using Selenium Grid?

selenium-node-chrome:103.0-chromedriver-103.0-grid-4.3.0-20220628

[github-actions[bot]]

@kongdengyuan, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[kongdengyuan]

@titusfortner Hello, Could you help me to take look this issue as soon as possible ? thanks

[diemol]

A few questions...

• Have you tried the most recent release (4.4.0)? Did it improve or is it the same?
• Was this fine in previous versions and it started to happen in the version you point out?
• How many CPUs can each running container access? If it is the whole 16, can you restrict to a max of 1 or 2 CPUs per container? What is the behavior when this change is done?

[kongdengyuan]

Thanks for your quick response. @diemol

• I used released 4.3.0 for our production env. and will use the latest version 4.4 to test performance .
• grid 3 and current 4.3 happened the same issue which memory leak , when is start node chrome docker , the selenium java process memory use from 200M to about 600M which no any task to call selenium process. After run test cases , the selenium java process may use memory around 800-1000M ,even if cases end the memory not release.
• This is memory issue not CPU , so I didn't limit CPU when I start docker containers , I already set JAVA_OPTS -Xmx384m to restrict java heap memory use and found that it can be reduce memory use, but I didn't know if it will affect the test results. So I still to researching.

[diemol]

@kongdengyuan the motivation behind limiting CPUs per container was because the HTTP client we use has a feature (or maybe a bug?) where a thread per CPU available is used. So if the container sees 16CPUs available, the Java process might end up using a bit too much memory. Can you please limit each container to 1CPU and see how that evolves?

[diemol]

Can you please try the new Java 11 HTTP client and let us know? https://www.selenium.dev/blog/2022/using-java11-httpclient/

[diemol]

I am going to close this since we did not get much more feedback.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.