python-driver icon indicating copy to clipboard operation
python-driver copied to clipboard

Memory leak during reconnection storm test

Open dkropachev opened this issue 6 months ago • 0 comments

While testing https://github.com/scylladb/python-driver/pull/473 I have found that in scenaior when driver can't properly initialize connections it start eating memory, in my case session endup consuming twice more memory than usual.

Repro: In order to test both PRs I have setup an environment with a cluster of 2 nodes, 16 smp each. And a swarm of client containers, each container holds 200 sessions.

Test scenario:

  1. Start cluster of 2 nodes, 16 smp each
  2. Populate the swarm of clients up to 30 clients: 2.1. Start with 0 containers 2.2. Grow the number of containers by 10 2.3. Wait till all shard connections are created 2.4. Repeat until reach target limit
  3. Restart one of the nodes
  4. Monitor number of connections in the swarm

You need to run it on scylladb/scylla:2025.2.1 with uninitialized_connections_semaphore_cpu_concurrency high enough to let scylla lag and fail connections during initialization.

Most-likely connections are leaking and stay in memory.

Repro code is attached. test.zip

dkropachev avatar Aug 07 '25 18:08 dkropachev