OSError: [Errno 9] Bad file descriptor

[leond08]

I'm having this issue in raspbian image

[2018-09-10 20:40:11 +0800] [21421] [CRITICAL] WORKER TIMEOUT (pid:21699) [2018-09-10 20:40:11 +0800] [21699] [ERROR] Socket error processing request. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/gunicorn/workers/async.py", line 62, in handle six.reraise(*sys.exc_info()) File "/usr/lib/python3/dist-packages/gunicorn/six.py", line 625, in reraise raise value File "/usr/lib/python3/dist-packages/gunicorn/workers/async.py", line 35, in handle listener_name = listener.getsockname() OSError: [Errno 9] Bad file descriptor

[benoitc]

@leond08 Thanks for the ticket!

In order to understand how it is happening can you provide a little more info?

• wich version of Gunicorn are you testing
• which worker are you using
• when does this happen?

[leond08]

I'm using gunicorn3 latest version I'm using eventlet and gevent for this i'm running my flask application - Flask-SocketIO

I start my background task after a user clicked the button My background task function is to listen to an event, after clicking the "DONE" button the background task must stop then send an emit message to all the users

[yunstanford]

Had the same issue with aiohttp + gunicorn, observe the same message each time when ctrl + c.

[INFO] Error while closing socket [Errno 9] Bad file descriptor

[benoitc]

I don't reproduce it. I suspect your application is closing some fds which create the issue above.

[dpzx]

We are experiencing the same issue, only thing is it's only happening on 1 out of 8 containers running in docker swarm.

[georgexsh]

We have experienced the same issue with 1 out of 9 containers, it looks like related to docker and python3 and gevent.

[zgoda]

gunicorn 20.0.4 + aiohttp 3.6.2

Gunicorn is running as dev server:

gunicorn --reload app:make_app --bind localhost:5000 --worker-class aiohttp.GunicornWebWorker --workers 2 --access-logfile -

Almost every Ctrl+C ends with

^C[2020-05-23 21:49:50 +0200] [38524] [INFO] Handling signal: int
Exception ignored when trying to write to the signal wakeup fd:
Exception ignored when trying to write to the signal wakeup fd:
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/unix_events.py", line 42, in _sighandler_noop
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/unix_events.py", line 42, in _sighandler_noop
    def _sighandler_noop(signum, frame):
    def _sighandler_noop(signum, frame):
OSError: [Errno 9] Bad file descriptor
OSError: [Errno 9] Bad file descriptor
[2020-05-23 21:49:50 +0200] [38526] [INFO] Worker exiting (pid: 38526)
[2020-05-23 21:49:50 +0200] [38528] [INFO] Worker exiting (pid: 38528)
[2020-05-23 21:49:50 +0200] [38524] [INFO] Shutting down: Master

It does not matter if application handled any request or not.

[zgoda]

With Sanic 20.3.0:

^C[2020-05-26 13:24:55 +0200] [27706] [INFO] Handling signal: int
[2020-05-26 13:24:55 +0200] [27769] [INFO] Stopping server: 27769, connections: 0
[2020-05-26 13:24:55 +0200] [27769] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-05-26 13:24:55 +0200] [27769] [INFO] Worker exiting (pid: 27769)
[2020-05-26 13:24:55 +0200] [27771] [INFO] Stopping server: 27771, connections: 0
[2020-05-26 13:24:55 +0200] [27771] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-05-26 13:24:55 +0200] [27771] [INFO] Worker exiting (pid: 27771)
[2020-05-26 13:24:55 +0200] [27706] [INFO] Shutting down: Master

[Soberia]

Same here with Gunicorn 20.0.4 + Uvicorn 0.11.5 worker class on every Ctrl+C

INFO:     [12621] [gunicorn.error] Handling signal: int
INFO:     [12635] [gunicorn.error] Error while closing socket [Errno 9] Bad file descriptor
INFO:     [12634] [gunicorn.error] Error while closing socket [Errno 9] Bad file descriptor
INFO:     [12635] [gunicorn.error] Worker exiting (pid: 12635)
INFO:     [12634] [gunicorn.error] Worker exiting (pid: 12634)
INFO:     [12621] [gunicorn.error] Shutting down: Master

[benoitc]

any example of application? Also which version of Python are we talking about?

[zgoda]

Ubuntu 20.04, system provided Python 3.8.2 in virtualenv

Example application: https://github.com/zgoda/newsloop-server/tree/d603a1c10c9e8be3d998f62ccc55dd73f4677115 (with aiohttp) or https://github.com/zgoda/newsloop-server/tree/b2a8a7f09fa9848d0384b51a3f6c0bb037fb459e (with Sanic). Exact gunicorn invocation in my earlier comment.

Differences in output between aiohttp and Sanic make me suspicious that there's something wrong involving workers.

[Ronserruya]

Same issue, python 3.8.0 sanic 19.12.2 gunicorn 20.0.4

Edit: this happens when i run locally on my Mac, but not when running inside a Linux docker, might help you

[strelkov-a]

Hello, I guess this issue https://github.com/benoitc/gunicorn/issues/2064 has the same reasons. We have almost the same errors like in issue, but we use gunicorn - 19.9.0

[fgimian]

Im experiencing this too, FastAPI + the latest Gunicorn and uvicorn workers with Python 3.8.5

As soon as I stop using uvicorn (i.e. remove this line from my gunicorn config):

worker_class = "uvicorn.workers.UvicornWorker"

The errors dissappear.

As described above, this happens when stopping Gunicorn with Ctrl+C or sending a graceful kill signal to the PID.

[2020-09-12 11:56:37 +1000] [100390] [INFO] Starting gunicorn 20.0.4
[2020-09-12 11:56:37 +1000] [100390] [INFO] Listening at: http://0.0.0.0:6000 (100390)
[2020-09-12 11:56:37 +1000] [100390] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-09-12 11:56:37 +1000] [100392] [INFO] Booting worker with pid: 100392
[2020-09-12 11:56:38 +1000] [100392] [INFO] Started server process [100392]
[2020-09-12 11:56:38 +1000] [100392] [INFO] Waiting for application startup.
[2020-09-12 11:56:38 +1000] [100392] [INFO] Application startup complete.
[2020-09-12 11:56:48 +1000] [100390] [INFO] Handling signal: term
[2020-09-12 11:56:48 +1000] [100392] [INFO] Shutting down
[2020-09-12 11:56:48 +1000] [100392] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-09-12 11:56:48 +1000] [100392] [INFO] Waiting for application shutdown.
[2020-09-12 11:56:48 +1000] [100392] [INFO] Application shutdown complete.
[2020-09-12 11:56:48 +1000] [100392] [INFO] Finished server process [100392]
[2020-09-12 11:56:48 +1000] [100392] [INFO] Worker exiting (pid: 100392)
[2020-09-12 11:56:48 +1000] [100390] [INFO] Shutting down: Master

[fgimian]

Here's an exact replication of the issue:

[fots@workstation testing]$ python3.8 -V
Python 3.8.5
[fots@workstation testing]$ python3.8 -m venv ~/.virtualenv/testing
[fots@workstation testing]$ source ~/.virtualenv/testing/bin/activate
(testing) [fots@workstation testing]$ pip install fastapi gunicorn uvicorn
Collecting fastapi
  Using cached fastapi-0.61.1-py3-none-any.whl (48 kB)
Collecting gunicorn
  Using cached gunicorn-20.0.4-py2.py3-none-any.whl (77 kB)
Collecting uvicorn
  Using cached uvicorn-0.11.8-py3-none-any.whl (43 kB)
Collecting pydantic<2.0.0,>=1.0.0
  Using cached pydantic-1.6.1-cp38-cp38-manylinux2014_x86_64.whl (11.5 MB)
Collecting starlette==0.13.6
  Using cached starlette-0.13.6-py3-none-any.whl (59 kB)
Requirement already satisfied: setuptools>=3.0 in /home/fots/.virtualenv/testing/lib/python3.8/site-packages (from gunicorn) (47.1.0)
Collecting h11<0.10,>=0.8
  Using cached h11-0.9.0-py2.py3-none-any.whl (53 kB)
Collecting websockets==8.*
  Using cached websockets-8.1-cp38-cp38-manylinux2010_x86_64.whl (78 kB)
Collecting httptools==0.1.*; sys_platform != "win32" and sys_platform != "cygwin" and platform_python_implementation != "PyPy"
  Using cached httptools-0.1.1-cp38-cp38-manylinux1_x86_64.whl (227 kB)
Collecting uvloop>=0.14.0; sys_platform != "win32" and sys_platform != "cygwin" and platform_python_implementation != "PyPy"
  Using cached uvloop-0.14.0-cp38-cp38-manylinux2010_x86_64.whl (4.7 MB)
Collecting click==7.*
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Installing collected packages: pydantic, starlette, fastapi, gunicorn, h11, websockets, httptools, uvloop, click, uvicorn
Successfully installed click-7.1.2 fastapi-0.61.1 gunicorn-20.0.4 h11-0.9.0 httptools-0.1.1 pydantic-1.6.1 starlette-0.13.6 uvicorn-0.11.8 uvloop-0.14.0 websockets-8.1
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/home/fots/.virtualenv/testing/bin/python3.8 -m pip install --upgrade pip' command.
(testing) [fots@workstation testing]$ ls -l
total 4
-rw-rw-r-- 1 fots fots 117 Sep 12 12:13 main.py
(testing) [fots@workstation testing]$ cat main.py
from fastapi import FastAPI

app = FastAPI()


@app.get("/")
async def root():
    return {"message": "Hello World"}
(testing) [fots@workstation testing]$ gunicorn -k uvicorn.workers.UvicornWorker main:app
[2020-09-12 12:19:05 +1000] [105788] [INFO] Starting gunicorn 20.0.4
[2020-09-12 12:19:05 +1000] [105788] [INFO] Listening at: http://127.0.0.1:8000 (105788)
[2020-09-12 12:19:05 +1000] [105788] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-09-12 12:19:05 +1000] [105790] [INFO] Booting worker with pid: 105790
[2020-09-12 12:19:05 +1000] [105790] [INFO] Started server process [105790]
[2020-09-12 12:19:05 +1000] [105790] [INFO] Waiting for application startup.
[2020-09-12 12:19:05 +1000] [105790] [INFO] Application startup complete.
^C[2020-09-12 12:19:06 +1000] [105788] [INFO] Handling signal: int
[2020-09-12 12:19:06 +1000] [105790] [INFO] Shutting down
[2020-09-12 12:19:06 +1000] [105790] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-09-12 12:19:06 +1000] [105790] [INFO] Waiting for application shutdown.
[2020-09-12 12:19:06 +1000] [105790] [INFO] Application shutdown complete.
[2020-09-12 12:19:06 +1000] [105790] [INFO] Finished server process [105790]
[2020-09-12 12:19:06 +1000] [105790] [INFO] Worker exiting (pid: 105790)
[2020-09-12 12:19:07 +1000] [105788] [INFO] Shutting down: Master

And here's the pip freeze output:

click==7.1.2
fastapi==0.61.1
gunicorn==20.0.4
h11==0.9.0
httptools==0.1.1
pydantic==1.6.1
starlette==0.13.6
uvicorn==0.11.8
uvloop==0.14.0
websockets==8.1

I attempted to install uvicorn and gunicorn from GitHub (master branch) to ensure I got the very latest fixes but the problem persisted.

Hope this helps Fotis

[gg349]

1. I am able to reproduce this log error with a setup with just gunicorn and uvicorn. This error message starts occurring with uvicorn==0.11.4 and not the previous version 0.11.3 (both on OSx and in a Linux container). This is consistent with the reports above with uvicorn, where the versions reported are alwasy greater than 0.11.4. Evidence at the end
2. The commit responsible for this error is this one. The problem lies with these few lines of the commit just mentioned. The commit changes only the order of two blocks of code. If I revert that change of order, the log error go away, while still passing the test suite of uvicorn
3. the same log error happens if one either: uses starlette and fastapi on top of the stack gunicorn+uvicorn as reported above; - runs the latest uvicorn version 12.X instead of 0.11.4; - runs gunicorn with more than just one uvicorn worker

Evidence. In a new folder on osx run the attached script test.sh (tested on osx). For testing in the Linux container, save both the script and the Dockerfile and then read the header of the Dockerfile. I also attach the log of the script.

@benoitc, what do you think of this commit in uvicorn that seems to introduce the bug? The problem seems to be at the interface between gunicorn and uvicorn. Can you comment on the ordering of the 2 blocks of code changed in the commit mentioned above in uvicorn? This may help finding out why this happens with the other cases too. So far this has been reported also with aiohttp, gevent, Flask-SocketIO sanic. I attach also the log of the script for ease.

log_test.log

file test.sh

#!/bin/bash
python3 -m venv venv
source venv/bin/activate
pip install gunicorn==20.0.04 uvicorn==0.11.4
# create simple uvicorn app
printf "async def app(scope, receive, send):\n    await send()\n" > example.py
# spin up gunicorn with 1 uvicorn worker, and then send TERM signal to gunicorn
gunicorn example:app -w 1 -k uvicorn.workers.UvicornWorker &
sleep 5 && pkill -f gunicorn
sleep 1
# you will see 1 log entry like this one:
# [XX] [YY] [INFO] Error while closing socket [Errno 9] Bad file descriptor

printf "\n\n[INFO] if you instead bump down uvicorn's version from 11.4 to 11.3 [Errno 9] goes away:\n\n"
pip install uvicorn==0.11.3
gunicorn example:app -w 1 -k uvicorn.workers.UvicornWorker &
sleep 5 && pkill -f gunicorn

file Dockerfile

# run with:
# docker run -it $(docker build -q .)
FROM python:3.8
COPY test.sh .
RUN chmod +x /test.sh
CMD /test.sh

[ChrisXiaoShu]

I had exactly same issue. Here is my case.

Brief : I'm trying to establish a test application for Django dwebsocket with gunicorn. When I trying to use websocket_client to test the result, after close the websocket this error happen everytime.

Environment : Docker image : python:3.7 python version : python3.7.6 gunicorn : version = 20.0.4, work = gevent Django version : Django==2.2 dwebsocket version : 0.5.12

Code:

view.py

from dwebsocket import accept_websocket

@accept_websocket
def my_ws(request):
    if request.is_websocket():
        ws = request.websocket
        while True:
            msg = ws.wait(timeout=15)
            if msg is None:
                print('get None message')
                break
            else:
                msg = b'echo :' + msg
                ws.send(msg)
                print('send ws seccess')
        print('websocket close')

urls.py

from websocketInfo.views import  my_ws
from django.conf.urls import url

urlpatterns = [
    url(r'my_ws/$', my_ws, name='my_ws')
]

websocket_client

from websocket import create_connection
ws = create_connection("ws://127.0.0.1:8080/my_ws/")
print("Sending 'Hello, World'...")
ws.send("Hello, World")
print("Receiving...")
result = ws.recv()
print(result)
ws.close()
print('ws close')

commad to run gunicorn server

gunicorn MyWebsocket.wsgi -b 0.0.0.0:8000 -w 3 -k gevent

error output

send ws seccess
get None message
websocket close
[2021-01-13 02:43:56 +0000] [101] [ERROR] Socket error processing request.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 65, in handle
    util.reraise(*sys.exc_info())
  File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 625, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 55, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
    super().handle_request(listener_name, req, sock, addr)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 128, in handle_request
    util.reraise(*sys.exc_info())
  File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 625, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 114, in handle_request
    resp.write(item)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 326, in write
    self.send_headers()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 322, in send_headers
    util.write(self.sock, util.to_bytestring(header_str, "latin-1"))
  File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 286, in write
    sock.sendall(data)
  File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 523, in sendall
    return _socketcommon._sendall(self, data_memory, flags)
  File "/usr/local/lib/python3.7/site-packages/gevent/_socketcommon.py", line 367, in _sendall
    chunk_size = max(socket.getsockopt(SOL_SOCKET, SO_SNDBUF), 1024 * 1024)
  File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 156, in __getattr__
    return getattr(self._sock, name)
  File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 66, in _dummy
    raise OSError(EBADF, 'Bad file descriptor')
OSError: [Errno 9] Bad file descriptor

[jamadden]

@ChrisXiaoShu The stack trace you posted tells us that this specific socket object has been explicitly closed at the Python level (that's when gevent uses its _dummy to generate the same exceptions that the operating system does). This means some part of your application stack is closing the socket before returning the response to let gunicorn handle it; at the point the error occurs, gunicorn hadn't even sent HTTP response headers yet.

[mawoka-myblock]

Kind of the same problem in my case, with the difference that this error happens without doing anything. Sometimes after 5min, sometimes after 2hrs...

[hardenchant]

had same issue on that setup:

Docker image : python:3.7.10-slim-stretch python version : python3.7.10 gunicorn : gunicorn = {version = "20.1.0", extras = ["eventlet"]} Django version : Django = "2.2.18"

[nazrulworld]

I don’t know if the issue is still unresolved! I had a similar issue (FastAPI + uvicorn worker + gunicorn + supervisor). Gunicorn was unable to stop workers and I saw the above error in the log.

I give the unique process name in the gunicorn settings proc_name and it's working!!

[jules-ch]

Last time I had this error It was because of a print statement, it raises this error & kill the worker.

Removing the print statement & replacing it with logging works. Hope it helps

[mindflayer]

I was trying to tackle this issue, since it's happening to us every time Gunicorn terminates an uvicorn worker which reached the max_requests limit. At some point of my research, I ended up reading https://bugs.python.org/issue29343 and I realised that maybe we should just forget about logging.info() in case of a socket.EBADF (Bad file descriptor). It's not an issue per se and some core devs are even proposing to remove this failure, just like it was before.

[mindflayer]

A second approach would be logging it as DEBUG instead of INFO. Let me know if you like any of the above, I could even submit a PR for it.

[nottrobin]

Similar problem here, k8s, gunicorn with gevent, flask:

OSError: [Errno 9] Bad file descriptor
2022-01-21 14:10:31.189Z ERROR gunicorn.error "Socket error processing request." service=ubuntu.com pid=12 errno=EBADF strerror="Bad file descriptor"
Traceback (most recent call last):
  File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 65, in handle
    util.reraise(*sys.exc_info())
  File "/root/.local/lib/python3.8/site-packages/gunicorn/util.py", line 626, in reraise
    raise value
  File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 38, in handle
    listener_name = listener.getsockname()
  File "/root/.local/lib/python3.8/site-packages/gevent/_socketcommon.py", line 557, in getsockname
    return self._sock.getsockname()
OSError: [Errno 9] Bad file descriptor
2022-01-21 14:10:31.190Z ERROR gunicorn.error "Socket error processing request." service=ubuntu.com pid=12 errno=EBADF strerror="Bad file descriptor"
Traceback (most recent call last):
  File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 65, in handle
    util.reraise(*sys.exc_info())
  File "/root/.local/lib/python3.8/site-packages/gunicorn/util.py", line 626, in reraise
    raise value
  File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 38, in handle
    listener_name = listener.getsockname()
  File "/root/.local/lib/python3.8/site-packages/gevent/_socketcommon.py", line 557, in getsockname
    return self._sock.getsockname()
OSError: [Errno 9] Bad file descriptor
2022-01-21 14:10:31.191Z ERROR gunicorn.error "Socket error processing request." service=ubuntu.com pid=12 errno=EBADF strerror="Bad file descriptor"
Traceback (most recent call last):
  File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 65, in handle
    util.reraise(*sys.exc_info())
  File "/root/.local/lib/python3.8/site-packages/gunicorn/util.py", line 626, in reraise
    raise value
  File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 38, in handle
    listener_name = listener.getsockname()
  File "/root/.local/lib/python3.8/site-packages/gevent/_socketcommon.py", line 557, in getsockname
    return self._sock.getsockname()
OSError: [Errno 9] Bad file descriptor
2022-01-21 14:10:31.191Z ERROR gunicorn.error "Socket error processing request." service=ubuntu.com pid=12 errno=EBADF strerror="Bad file descriptor"
Traceback (most recent call last):
  File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 65, in handle
    util.reraise(*sys.exc_info())
  File "/root/.local/lib/python3.8/site-packages/gunicorn/util.py", line 626, in reraise
    raise value
  File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 38, in handle
    listener_name = listener.getsockname()
  File "/root/.local/lib/python3.8/site-packages/gevent/_socketcommon.py", line 557, in getsockname
    return self._sock.getsockname()
OSError: [Errno 9] Bad file descriptor
2022-01-21 14:10:31.195Z ERROR flask.app "Exception on /security/cve [GET]" service=ubuntu.com pid=12 request_id=e84ae25c8bc81046044c52901f176248
Traceback (most recent call last):
  File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/root/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/srv/webapp/security/views.py", line 735, in cve_index
    return flask.render_template(
  File "/root/.local/lib/python3.8/site-packages/flask/templating.py", line 136, in render_template
    ctx.app.update_template_context(context)
  File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 838, in update_template_context
    context.update(func())
  File "/srv/webapp/app.py", line 272, in context
    "releases": releases(),
  File "/srv/webapp/context.py", line 39, in releases
    return yaml.load(releases, Loader=yaml.FullLoader)
  File "/root/.local/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/root/.local/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 71, in compose_node
    event = self.peek_event()
  File "/root/.local/lib/python3.8/site-packages/yaml/parser.py", line 107, in peek_event
    def peek_event(self):
  File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 202, in handle_abort
    self.cfg.worker_abort(self)
  File "/root/.local/lib/python3.8/site-packages/talisker/gunicorn.py", line 123, in gunicorn_worker_abort
    raise talisker.wsgi.RequestTimeout(
talisker.wsgi.RequestTimeout: gunicorn worker timeout (pid: 12)
2022-01-21 14:10:31.997Z INFO talisker.wsgi "GET /security/cve?" method=GET path=/security/cve qs="package=libcurl4" status=500 view=webapp.security.views.cve_index duration_ms=9994.17 ip=91.189.88.181 proto=HTTP/1.1 length=9295 forwarded="2600:1900:2000:d9::2, 91.189.88.181" ua=Go-http-client/1.1 service=ubuntu.com pid=12 request_id=e84ae25c8bc81046044c52901f176248
2022-01-21 14:10:34.043Z INFO gunicorn.error "Worker exiting (pid: 12)" service=ubuntu.com pid=12
Sentry is attempting to send 126 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
2022-01-21 14:10:44.956Z INFO gunicorn.error "Booting worker with pid: 41"
2022-01-21 14:10:47.595Z INFO talisker.flask "updating raven config from flask app"

It sounds kind of like some people think this is a benign error that shouldn't be logged, but in our case it appears to be killing the worker - so not benign.

[mindflayer]

It sounds kind of like some people think this is a benign error that shouldn't be logged, but in our case it appears to be killing the worker - so not benign.

That's definitely an issue, but Gunicorn logs the same error as INFO logs somewhere else, and that's the one I was referring to. https://github.com/benoitc/gunicorn/blob/cf55d2cec277f220ebd605989ce78ad1bb553c46/gunicorn/sock.py#L69

[onekiloparsec]

I am surprised to see this issue still open. Following the report from @gg349 (https://github.com/benoitc/gunicorn/issues/1877#issuecomment-739526302) above, I've made a fork and inverted the lines here:

https://github.com/encode/uvicorn/commit/c4900d19e1100a7b1a93a99f3d3ec6b717ffea41#diff-ab3f29e794a752a9514c33ce772320a2cfbeab571f5254207f096e2d8da7e077L530-R533

And I run smoothly since months... I do not clearly understand the impact of this change, hence I didn't make a PR. But I am now reluctant of going back to the official branch.

[matteius]

This seems to be biting me as well running saleor API deployed with gunicorn + uvicorn worker. The commit from @onekiloparsec kind of makes sense but that commit branch is old enough that its not compatible with my other dependencies when I try to install it. Forking the uvicorn repo to try and cherry pick that commit into my running version to see if it helps.

EDIT: Actually the 0.17.6 branch has the code already in the form of @onekiloparsec commit. I perhaps started seeing this only after adding --max-requests 2 argument to gunicorn.

Latest uvicorn logic:

        # Stop accepting new connections.
        for server in self.servers:
            server.close()
        for sock in sockets or []:
            sock.close()
        for server in self.servers:
            await server.wait_closed()

[onekiloparsec]

@matteius if you confirm running smoothly for a few days with 0.17.6, I may reconsider going back to the main branch!

[benoitc]

i am not reproucingi t myself. closing the issue