Unable to start up docker stack past v1.92.2

[AetherUnbound]

trafficstars

Description

When upgrading, it doesn't look like we're able to upgrade pass v1.92.1, since v1.92.2 is now using debian bookworm. Under bookworm, we're seeing this issue when we try to run the stack: https://github.com/docker-library/python/issues/835

Would it be possible to add a -bullseye specific docker image release for newer versions, until the upstream Docker issue is resolved?

Steps to reproduce

(per this comment)

Docker version: 20.10.5 runc version: 1.0.0-rc93 libseccomp2: 2.5.1-1ubuntu1~20.04.2 Matrix version attempted: v1.97.0

Homeserver

matrix.tech-bloc-sea.dev

Synapse Version

1.92.1

Installation Method

Docker (matrixdotorg/synapse)

Database

Postgres, single server

Workers

Single process

Platform

Ubuntu 20.04.6

Configuration

No response

Relevant log output

synapse       | 2023-12-01 20:56:17,771 - synapse.app._base - 267 - CRITICAL - sentinel - Error during startup
synapse       | Traceback (most recent call last):
synapse       |   File "/usr/local/lib/python3.11/site-packages/synapse/app/_base.py", line 252, in wrapper
synapse       |     await cb(*args, **kwargs)
synapse       |   File "/usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py", line 364, in start
synapse       |     await _base.start(hs)
synapse       |   File "/usr/local/lib/python3.11/site-packages/synapse/app/_base.py", line 526, in start
synapse       |     resolver_threadpool.start()
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 178, in start
synapse       |     self.adjustPoolsize()
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 331, in adjustPoolsize
synapse       |     self._team.grow(self.min - self.workers)
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_team.py", line 126, in grow
synapse       |     @self._coordinator.do
synapse       |      ^^^^^^^^^^^^^^^^^^^^
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_threadworker.py", line 109, in do
synapse       |     working.pop(0)()
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_team.py", line 129, in createOneWorker
synapse       |     worker = self._createWorker()
synapse       |              ^^^^^^^^^^^^^^^^^^^^
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_pool.py", line 66, in limitedWorkerCreator
synapse       |     return ThreadWorker(startThread, Queue())
synapse       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_threadworker.py", line 51, in __init__
synapse       |     startThread(work)
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_pool.py", line 60, in startThread
synapse       |     return threadFactory(target=target).start()
synapse       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
synapse       |   File "/usr/local/lib/python3.11/threading.py", line 964, in start
synapse       |     _start_new_thread(self._bootstrap, ())
synapse       | RuntimeError: can't start new thread
synapse       | Error during startup:
synapse       | Traceback (most recent call last):
synapse       |   File "/usr/local/lib/python3.11/site-packages/synapse/app/_base.py", line 252, in wrapper
synapse       |     await cb(*args, **kwargs)
synapse       |   File "/usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py", line 364, in start
synapse       |     await _base.start(hs)
synapse       |   File "/usr/local/lib/python3.11/site-packages/synapse/app/_base.py", line 526, in start
synapse       |     resolver_threadpool.start()
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 178, in start
synapse       |     self.adjustPoolsize()
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 331, in adjustPoolsize
synapse       |     self._team.grow(self.min - self.workers)
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_team.py", line 126, in grow
synapse       |     @self._coordinator.do
synapse       |      ^^^^^^^^^^^^^^^^^^^^
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_threadworker.py", line 109, in do
synapse       |     working.pop(0)()
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_team.py", line 129, in createOneWorker
synapse       |     worker = self._createWorker()
synapse       |              ^^^^^^^^^^^^^^^^^^^^
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_pool.py", line 66, in limitedWorkerCreator
synapse       |     return ThreadWorker(startThread, Queue())
synapse       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_threadworker.py", line 51, in __init__
synapse       |     startThread(work)
synapse       |   File "/usr/local/lib/python3.11/site-packages/twisted/_threads/_pool.py", line 60, in startThread
synapse       |     return threadFactory(target=target).start()
synapse       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
synapse       |   File "/usr/local/lib/python3.11/threading.py", line 964, in start
synapse       |     _start_new_thread(self._bootstrap, ())
synapse       | RuntimeError: can't start new thread

Anything else that would be useful to know?

We saw this exact issue with another one of our service: https://github.com/OrcaCollective/OpenOversight/issues/332

[DMRobertson]

Would it be possible to add a -bullseye specific docker image release for newer versions, until the upstream Docker issue is resolved?

AFAICS bullseye is still supported by the Debian project (https://endoflife.date/debian) and the docker container also supports the bullseye tag (https://hub.docker.com/_/debian). On the face of it, it looks possible. @erikjohnston what was the motivation for using bookworm in #16324?

Docker version: 20.10.5

This blog (which I found from here, in turn from here) suggests that upgrading Docker to 20.10.10 on the host may help. Those links are from two years ago, referring to glibc 2.34. From here it looks like the bullseye->bookworm transition moved from glibc 2.31 to 2.36, so it seems plausible that updating docker on your host would work here. Is this something you're able to try?

[AetherUnbound]

It definitely is, we will give that a shot and report back!

[AetherUnbound]

Huh, I guess we're already at the latest version 😕

$ sudo apt upgrade docker-ce
Reading package lists... Done
Building dependency tree       
Reading state information... Done
docker-ce is already the newest version (5:20.10.5~3-0~ubuntu-focal).