RuntimeError: can't start new thread (0.23.2)

[amolkavitkar]

Long story short

operator pod gets killed due to hitting thread limit reached as process keeps on spawning new threads and in loads condition pod gets killed in every 2 Hrs

Description

RuntimeError: can't start new thread Traceback (most recent call last): File "/usr/local/bin/kopf", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in call return self.main(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/kopf/cli.py", line 30, in wrapper return fn(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/kopf/cli.py", line 61, in run peering_name=peering_name, File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 275, in run _reraise(loop, list(done1) + list(done2) + list(done3) + list(done4)) File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 303, in _reraise task.result() # can raise the regular (non-cancellation) exceptions. File "/usr/local/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 81, in watcher async for event in watching.infinite_watch(resource=resource, namespace=namespace): File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 131, in infinite_watch async for event in streaming_watch(resource=resource, namespace=namespace): File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 93, in streaming_watch async for event in streaming_aiter(stream, loop=loop): File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 62, in streaming_aiter yield await loop.run_in_executor(executor, streaming_next, src) File "/usr/local/lib/python3.7/asyncio/base_events.py", line 747, in run_in_executor executor.submit(func, *args), loop=self) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 172, in submit self._adjust_thread_count() File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 193, in _adjust_thread_count t.start() File "/usr/local/lib/python3.7/threading.py", line 852, in start _start_new_thread(self._bootstrap, ())

root@s/# ps huH p 8 | wc -l 624 root@s:/# ps -o nlwp 8 NLWP 624 root@s:/# ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 08:46 ? 00:00:00 /bin/sh -c kopf run --standalone /handlers.py root 8 1 0 08:46 ? 00:00:13 /usr/local/bin/python /usr/local/bin/kopf run --standalone /handlers.py root 634 0 0 11:00 pts/0 00:00:00 bash root 649 634 0 11:01 pts/0 00:00:00 ps -ef root@serviceendpoint-6b69949674-tvbj6:/# cat /proc/sys/kernel/threads-max 6180721 root@s:/# ps -o nlwp 8 NLWP 631 root@s:/# ps -o nlwp 8 NLWP 652

import kopf

The exact command to reproduce the issue

kopf run ...

The full output of the command that failed

Environment

• Kopf version: 0.23.2
• Kubernetes version: 1.15.3
• Python version: 3.7
• OS/platform: ubuntu 16.04

Python packages installed

[nolar]

Hello. Thanks for reporting.

Can you please clarify the version? Are you sure it is version 0.23.2?

I see this line:

File "/usr/local/lib/python3.7/site-packages/kopf/clients/watching.py", line 62, in streaming_aiter
yield await loop.run_in_executor(executor, streaming_next, src)

This sync approach was removed in #227 (line link), which was released as 0.23 (followed by 0.23.1 & 0.23.2). — And it was replaced with aiohttp-based cycles, which is natively async — and remains so since 0.23+.

[nolar]

One idea how this happens:

Kopf uses a threaded executor of asyncio for synchronous handlers (declared with def, not async def).

Python's threaded executor adds one more thread on every use until max_workers is reached (link) — not all threads at once, but one-by-one.

The default max_workers is os.cpu_count() * 5 (link). So, on a MacBook, it can be 8 * 5 = 40 (perhaps, due to hyperthreading CPUs). On huge K8s nodes, it can be up to 40 * 5 = 200 threads (assuming 40 regular cores), or 2 * 40 * 5 = 400 (40 hyper-threading cores), or more and more.

It keeps the threaded workers running even if they are not used. And it does not reuse the idle workers for the next tasks (or maybe it does, but still adds extra idling workers until the limit is reached).

Maybe, at some level, it reaches the RAM limits of the pod and dies.

This can be controlled by using an already existing but undocumented config (link):

import kopf

kopf.config.WorkersConfig.set_synchronous_tasks_threadpool_limit(100)

I was able to catch this on one of our operators which is supposed to do nothing during that time (but contained synchronous handlers for @kopf.on.event of pods) — and the thread count was growing overnight when it should be flat. I will try this trick above and see if it helps in the next few nights.

[nolar]

It did help. The thread count does not grow once the operator is started and used: it stays at 12 (10 were configured for the executor, 1 for the main thread, 1 for something Python's perhaps).

So far, the issue is a matter of asyncio's thread executor setup — can be done on the operator level as needed. I suggest that we have no own defaults or assumptions about the execution environment in the framework.

Tested with Kopf 0.25.

@amolkavitkar Can you please check if this solutions helps you? (Also, please check the version.)