gunicorn worker (gevent) failed to boot using python 3.11

[PavanTatikonda]

We are using alpine 3.16.2, python 3.11.0 and pip 22.3 along with bunch of other libraries (django, google, etc.) installing via pip

gunicorn==20.1.0 gevent==22.10.2 greenlet==2.0.1

gunicorn worker (gevent) failed to boot

[2022-11-30 09:19:27 +0000] [6] [INFO] Starting gunicorn 20.1.0
[2022-11-30 09:19:27 +0000] [6] [INFO] Listening at: https://0.0.0.0:8443 (6)
[2022-11-30 09:19:27 +0000] [6] [INFO] Using worker: gevent
[2022-11-30 09:19:27 +0000] [7] [INFO] Booting worker with pid: 7
[2022-11-30 09:19:28 +0000] [8] [INFO] Booting worker with pid: 8
[2022-11-30 09:19:29 +0000] [7] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/ggevent.py", line 144, in init_process
    self.patch()
  File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/ggevent.py", line 38, in patch
    monkey.patch_all()
  File "/usr/local/lib/python3.11/site-packages/gevent/monkey.py", line 1263, in patch_all
    patch_thread(Event=Event, _warnings=_warnings)
  File "/usr/local/lib/python3.11/site-packages/gevent/monkey.py", line 200, in ignores
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/gevent/monkey.py", line 791, in patch_thread
    _patch_existing_locks(threading_mod)
  File "/usr/local/lib/python3.11/site-packages/gevent/monkey.py", line 672, in _patch_existing_locks
    if isinstance(o, rlock_type):
       ^^^^^^^^^^^^^^^^^^^^^^^^^
ReferenceError: weakly-referenced object no longer exists
[2022-11-30 09:19:29 +0000] [7] [INFO] Worker exiting (pid: 7)
[2022-11-30 09:19:29 +0000] [8] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/ggevent.py", line 144, in init_process
    self.patch()
  File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/ggevent.py", line 38, in patch
    monkey.patch_all()
  File "/usr/local/lib/python3.11/site-packages/gevent/monkey.py", line 1263, in patch_all
    patch_thread(Event=Event, _warnings=_warnings)
  File "/usr/local/lib/python3.11/site-packages/gevent/monkey.py", line 200, in ignores
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/gevent/monkey.py", line 791, in patch_thread
    _patch_existing_locks(threading_mod)
  File "/usr/local/lib/python3.11/site-packages/gevent/monkey.py", line 672, in _patch_existing_locks
    if isinstance(o, rlock_type):
       ^^^^^^^^^^^^^^^^^^^^^^^^^
ReferenceError: weakly-referenced object no longer exists
[2022-11-30 09:19:29 +0000] [8] [INFO] Worker exiting (pid: 8)
[2022-11-30 09:19:30 +0000] [6] [WARNING] Worker with pid 7 was terminated due to signal 15
[2022-11-30 09:19:30 +0000] [6] [INFO] Shutting down: Master
[2022-11-30 09:19:30 +0000] [6] [INFO] Reason: Worker failed to boot.

Same is working fine using alpine 3.16.2, python 3.10.7 and pip 22.2.2 along with same bunch of other libraries (django, google, etc.) installing via pip

Could someone please help on this and please let me know if any other information is needed ?

Note:- Any insights on how to triage/debug this further is also helpful

[frosk1]

Hey, any update on when python 3.11 will be supported?

[jim-schwoebel]

having this issue as well, can reproduce on python v 3.11.6

[justinborromeo-glean]

Having this issue on python 3.10.14

[Obchelli]

I have similar issue python3.11.9

[benoitc]

error looks relared to gevent. @Obchelli which versions of gunicorn / gevent are you using?

[jonnor]

We are seeing the same exception on Python 3.12.5, using Docker images based on python:3.12.5-bookworm. An odd thing is that we experience the problem when built on Heroku, whereas when we build/run the same Dockerfiles on Gitlab CI - there is no such exception.

Using gevent==24.2.1 and gunicorn==21.2.0 now. Issue also happened with gevent==23.9.1 and gunicorn==20.1.0

Would love some tips on what information to collect in order to debug/fix this issue.

[jonnor]

In our case, we found that the issue was triggered by New Relic, in particular using the newrelic-admin run-command instrumentation for our Python application. Removing this made the issue disappear. We found this via https://stackoverflow.com/a/75665547/1967571 - which looks to be reported by the same person as this issue originally - @PavanTatikonda