uvicorn
uvicorn copied to clipboard
Published
20 hours ago •
encode
encode
Test workers
Description
The uvicorn.workers module provides worker classes intended for use as Gunicorn workers. The Uvicorn docs say, "For production deployments we recommend using gunicorn with the uvicorn worker class." Other projects suggest similar use cases. For example, Sanic suggests running Gunicorn with the Uvicorn worker (by exposing the Sanic app instance as an ASGI app to Uvicorn).
Although we recommend running workers in production, we don't have tests for this approach. It would be helpful to test the workers if we're recommending them for production deployments. PR #631 added some tests for the Gunicorn worker, but these tests were deemed "flaky" (unclear why) and removed in #650 and #658. Gunicorn doesn't really test its own workers either (benoitc/gunicorn#2742), which gives us even more reason to run tests in this project.
Changes
Worker tests
This PR offers tests for 100% of the Gunicorn worker code in a new module tests/test_workers.py. Test fixtures are also stored in tests/test_workers.py, rather than in tests/conftest.py, to cleanly separate code depending on Gunicorn from the rest of the tests.
A test fixture starts a subprocess running Gunicorn with a Uvicorn worker and an ASGI app. The subprocess includes an instance of httpx.Client for HTTP requests to the Uvicorn worker's ASGI app, and saves its output to a temporary file for assertions on stdout/stderr. Tests can send signals to the process.
Coverage configuration
Previously, uvicorn.workers was completely omitted from coverage measurement due to use of the include setting to specify source files. Uvicorn recently switched the coverage.py config from the include setting to the source_pkgs setting, which shows the expected coverage (https://github.com/encode/uvicorn/pull/1990#discussion_r1213446062).
This PR will add tests to cover 100% of the worker code by configuring coverage.py for subprocess test coverage measurement. There is some longstanding awkwardness involved (nedbat/coveragepy#367). Changes in this PR include:
- Enable the required parallel mode
- Set the required
COVERAGE_PROCESS_STARTenvironment variable - Add the
coverage_enable_subprocesspackage to invokecoverage.process_startup - Combine coverage reports before reporting coverage
- Update
.gitignoreto ignore files named.coverage*(many coverage files are generated when subprocesses are measured in parallel mode)
Checklist
- [x] I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
- [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
- [x] I've updated the documentation accordingly.
Related
Uvicorn
- Closes encode/uvicorn#1834
- https://github.com/encode/uvicorn/issues/102#issuecomment-471904183
- https://github.com/encode/uvicorn/pull/631 (PR 631 added
tests/test_gunicorn_workers.py, reverted by https://github.com/encode/uvicorn/pull/650 and https://github.com/encode/uvicorn/pull/658) - https://github.com/encode/uvicorn/pull/947#discussion_r833717485
- Future work:
- It would be helpful to test Uvicorn worker
SIGQUITsignal handling. When aSIGINTis sent to the Gunicorn process, Gunicorn sendsSIGQUITto the Uvicorn worker instead ofSIGINT, and the Uvicorn worker runshandle_exitinstead ofhandle_quit(benoitc/gunicorn#2604, encode/uvicorn#1116). ASIGQUIThandler was added in #1710, but this behavior has not been tested. - There's been some interest in moving the workers to a separate package (https://github.com/encode/uvicorn/issues/517#issuecomment-564090865, https://github.com/encode/uvicorn/pull/1205). If we end up doing this, I'm happy to contribute to the new package as well.
- It would be helpful to test Uvicorn worker
Other projects
- https://github.com/benoitc/gunicorn/issues/2742
- PrefectHQ/prefect#7948 (instructive example of server testing fixtures and signal handling tests)
- wemake-services/coverage-conditional-plugin#221 (support for omitting modules in coverage-conditional-plugin)
- nedbat/coveragepy#367
- Coverage docs: Measuring subprocesses
- Sanic 22.3 (Sanic used to have a Gunicorn worker, but they removed it in 22.3, and they now suggest running Gunicorn with the Uvicorn worker)
- Sanic docs: User Guide - Deployment - Worker Manager
Help wanted
I'm seeing some occasional test flakes with tests/protocols/test_websocket.py::test_connection_lost_before_handshake_complete. This PR is not modifying any of the websocket code, so I'm not sure why these flakes are occurring.
The tests sometimes fail on this line:
https://github.com/encode/uvicorn/blob/0624d5732cf5d0f3fe73657550da7d0242510d94/tests/protocols/test_websocket.py#L754
The errors typically look like this (expand).
Run scripts/test
+ [ -z true ]
+ export COVERAGE_PROCESS_START=/home/runner/work/uvicorn/uvicorn/pyproject.toml
+ coverage run --debug config -m pytest
-- config ----------------------------------------------------
attempted_config_files: .coveragerc
setup.cfg
tox.ini
pyproject.toml
branch: False
command_line: None
concurrency: -none-
config_file: /home/runner/work/uvicorn/uvicorn/pyproject.toml
config_files_read: /home/runner/work/uvicorn/uvicorn/pyproject.toml
context: None
cover_pylib: False
data_file: .coverage
debug: config
debug_file: None
disable_warnings: -none-
dynamic_context: None
exclude_also: -none-
exclude_list: pragma: no cover
pragma: nocover
if TYPE_CHECKING:
if typing.TYPE_CHECKING:
raise NotImplementedError
py-not-win32
py-linux
py-gte-38
extra_css: None
fail_under: 98.05
format: None
html_dir: htmlcov
html_skip_covered: None
html_skip_empty: None
html_title: Coverage report
ignore_errors: False
include_namespace_packages: False
json_output: coverage.json
json_pretty_print: False
json_show_contexts: False
lcov_output: coverage.lcov
parallel: True
partial_always_list: while (True|1|False|0):
if (True|1|False|0):
partial_list: #\s*(pragma|PRAGMA)[:\s]?\s*(no|NO)\s*(branch|BRANCH)
paths: {}
plugin_options: {'coverage_conditional_plugin': {'omit': {"sys_platform == 'win32'": ['tests/test_workers.py', 'uvicorn/workers.py']}, 'rules': {'py-win32': "sys_platform == 'win32'", 'py-not-win32': "sys_platform != 'win32'", 'py-linux': "sys_platform == 'linux'", 'py-darwin': "sys_platform == 'darwin'", 'py-gte-38': 'sys_version_info >= (3, 8)', 'py-lt-38': 'sys_version_info < (3, 8)', 'py-gte-39': 'sys_version_info < (3, 9)', 'py-lt-39': 'sys_version_info < (3, 9)'}}}
plugins: coverage_conditional_plugin
precision: 2
relative_files: False
report_contexts: None
report_include: None
report_omit: None
run_include: -none-
run_omit: -none-
show_contexts: False
show_missing: True
sigterm: False
skip_covered: True
skip_empty: False
sort: None
source: None
source_pkgs: uvicorn
tests
timid: False
xml_output: coverage.xml
xml_package_depth: 99
-- end -------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.10.11, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/runner/work/uvicorn/uvicorn, configfile: pyproject.toml
plugins: mock-3.10.0, anyio-3.7.0
collected 565 items
tests/test_auto_detection.py ... [ 0%]
tests/test_cli.py ............ [ 2%]
tests/test_config.py ................................................... [ 11%]
....................................................... [ 21%]
tests/test_default_headers.py ...... [ 22%]
tests/test_lifespan.py ................ [ 25%]
tests/test_main.py ......... [ 26%]
tests/test_ssl.py .... [ 27%]
tests/test_subprocess.py .. [ 27%]
tests/test_workers.py .......................................... [ 35%]
tests/importer/test_importer.py ...... [ 36%]
tests/middleware/test_logging.py .............. [ 38%]
tests/middleware/test_message_logger.py .. [ 39%]
tests/middleware/test_proxy_headers.py ........... [ 41%]
tests/middleware/test_wsgi.py ........... [ 43%]
tests/protocols/test_http.py ........................................... [ 50%]
................................................................... [ 62%]
tests/protocols/test_utils.py ...... [ 63%]
tests/protocols/test_websocket.py ...................................... [ 70%]
........................................................................ [ 83%]
......FFFF.............................................. [ 93%]
tests/supervisors/test_multiprocess.py . [ 93%]
tests/supervisors/test_reload.py ................................... [ 99%]
tests/supervisors/test_signal.py ... [100%]
=================================== FAILURES ===================================
____ test_connection_lost_before_handshake_complete[H11Protocol-WSProtocol] ____
ws_protocol_cls = <class 'uvicorn.protocols.websockets.wsproto_impl.WSProtocol'>
http_protocol_cls = <class 'uvicorn.protocols.http.h11_impl.H11Protocol'>
unused_tcp_port = 39059
@pytest.mark.anyio
@pytest.mark.parametrize("ws_protocol_cls", WS_PROTOCOLS)
@pytest.mark.parametrize("http_protocol_cls", HTTP_PROTOCOLS)
async def test_connection_lost_before_handshake_complete(
ws_protocol_cls, http_protocol_cls, unused_tcp_port: int
):
send_accept_task = asyncio.Event()
disconnect_message = {}
async def app(scope, receive, send):
nonlocal disconnect_message
message = await receive()
if message["type"] == "websocket.connect":
await send_accept_task.wait()
disconnect_message = await receive()
response: typing.Optional[httpx.Response] = None
async def websocket_session(uri):
nonlocal response
async with httpx.AsyncClient() as client:
response = await client.get(
f"http://127.0.0.1:{unused_tcp_port}",
headers={
"upgrade": "websocket",
"connection": "upgrade",
"sec-websocket-version": "13",
"sec-websocket-key": "dGhlIHNhbXBsZSBub25jZQ==",
},
)
config = Config(
app=app,
ws=ws_protocol_cls,
http=http_protocol_cls,
lifespan="off",
port=unused_tcp_port,
)
async with run_server(config):
task = asyncio.create_task(
websocket_session(f"ws://127.0.0.1:{unused_tcp_port}")
)
await asyncio.sleep(0.1)
send_accept_task.set()
task.cancel()
> assert response is not None
E assert None is not None
tests/protocols/test_websocket.py:754: AssertionError
----------------------------- Captured stderr call -----------------------------
INFO: Started server process [2037]
INFO: Uvicorn running on http://127.0.0.1:39059 (Press CTRL+C to quit)
INFO: Shutting down
------------------------------ Captured log call -------------------------------
INFO uvicorn.error:server.py:76 Started server process [2037]
INFO uvicorn.error:server.py:219 Uvicorn running on http://127.0.0.1:39059 (Press CTRL+C to quit)
INFO uvicorn.error:server.py:265 Shutting down
_ test_connection_lost_before_handshake_complete[H11Protocol-WebSocketProtocol] _
ws_protocol_cls = <class 'uvicorn.protocols.websockets.websockets_impl.WebSocketProtocol'>
http_protocol_cls = <class 'uvicorn.protocols.http.h11_impl.H11Protocol'>
unused_tcp_port = 35627
@pytest.mark.anyio
@pytest.mark.parametrize("ws_protocol_cls", WS_PROTOCOLS)
@pytest.mark.parametrize("http_protocol_cls", HTTP_PROTOCOLS)
async def test_connection_lost_before_handshake_complete(
ws_protocol_cls, http_protocol_cls, unused_tcp_port: int
):
send_accept_task = asyncio.Event()
disconnect_message = {}
async def app(scope, receive, send):
nonlocal disconnect_message
message = await receive()
if message["type"] == "websocket.connect":
await send_accept_task.wait()
disconnect_message = await receive()
response: typing.Optional[httpx.Response] = None
async def websocket_session(uri):
nonlocal response
async with httpx.AsyncClient() as client:
response = await client.get(
f"http://127.0.0.1:{unused_tcp_port}",
headers={
"upgrade": "websocket",
"connection": "upgrade",
"sec-websocket-version": "13",
"sec-websocket-key": "dGhlIHNhbXBsZSBub25jZQ==",
},
)
config = Config(
app=app,
ws=ws_protocol_cls,
http=http_protocol_cls,
lifespan="off",
port=unused_tcp_port,
)
async with run_server(config):
task = asyncio.create_task(
websocket_session(f"ws://127.0.0.1:{unused_tcp_port}")
)
await asyncio.sleep(0.1)
send_accept_task.set()
task.cancel()
> assert response is not None
E assert None is not None
tests/protocols/test_websocket.py:754: AssertionError
----------------------------- Captured stderr call -----------------------------
INFO: Started server process [2037]
INFO: Uvicorn running on http://127.0.0.1:35627 (Press CTRL+C to quit)
INFO: Shutting down
------------------------------ Captured log call -------------------------------
INFO uvicorn.error:server.py:76 Started server process [2037]
INFO uvicorn.error:server.py:219 Uvicorn running on http://127.0.0.1:35627 (Press CTRL+C to quit)
INFO uvicorn.error:server.py:265 Shutting down
_ test_connection_lost_before_handshake_complete[HttpToolsProtocol-WSProtocol] _
ws_protocol_cls = <class 'uvicorn.protocols.websockets.wsproto_impl.WSProtocol'>
http_protocol_cls = <class 'uvicorn.protocols.http.httptools_impl.HttpToolsProtocol'>
unused_tcp_port = 56383
@pytest.mark.anyio
@pytest.mark.parametrize("ws_protocol_cls", WS_PROTOCOLS)
@pytest.mark.parametrize("http_protocol_cls", HTTP_PROTOCOLS)
async def test_connection_lost_before_handshake_complete(
ws_protocol_cls, http_protocol_cls, unused_tcp_port: int
):
send_accept_task = asyncio.Event()
disconnect_message = {}
async def app(scope, receive, send):
nonlocal disconnect_message
message = await receive()
if message["type"] == "websocket.connect":
await send_accept_task.wait()
disconnect_message = await receive()
response: typing.Optional[httpx.Response] = None
async def websocket_session(uri):
nonlocal response
async with httpx.AsyncClient() as client:
response = await client.get(
f"http://127.0.0.1:{unused_tcp_port}",
headers={
"upgrade": "websocket",
"connection": "upgrade",
"sec-websocket-version": "13",
"sec-websocket-key": "dGhlIHNhbXBsZSBub25jZQ==",
},
)
config = Config(
app=app,
ws=ws_protocol_cls,
http=http_protocol_cls,
lifespan="off",
port=unused_tcp_port,
)
async with run_server(config):
task = asyncio.create_task(
websocket_session(f"ws://127.0.0.1:{unused_tcp_port}")
)
await asyncio.sleep(0.1)
send_accept_task.set()
task.cancel()
> assert response is not None
E assert None is not None
tests/protocols/test_websocket.py:754: AssertionError
----------------------------- Captured stderr call -----------------------------
INFO: Started server process [2037]
INFO: Uvicorn running on http://127.0.0.1:56383 (Press CTRL+C to quit)
INFO: Shutting down
------------------------------ Captured log call -------------------------------
INFO uvicorn.error:server.py:76 Started server process [2037]
INFO uvicorn.error:server.py:219 Uvicorn running on http://127.0.0.1:56383 (Press CTRL+C to quit)
INFO uvicorn.error:server.py:265 Shutting down
_ test_connection_lost_before_handshake_complete[HttpToolsProtocol-WebSocketProtocol] _
ws_protocol_cls = <class 'uvicorn.protocols.websockets.websockets_impl.WebSocketProtocol'>
http_protocol_cls = <class 'uvicorn.protocols.http.httptools_impl.HttpToolsProtocol'>
unused_tcp_port = 33179
@pytest.mark.anyio
@pytest.mark.parametrize("ws_protocol_cls", WS_PROTOCOLS)
@pytest.mark.parametrize("http_protocol_cls", HTTP_PROTOCOLS)
async def test_connection_lost_before_handshake_complete(
ws_protocol_cls, http_protocol_cls, unused_tcp_port: int
):
send_accept_task = asyncio.Event()
disconnect_message = {}
async def app(scope, receive, send):
nonlocal disconnect_message
message = await receive()
if message["type"] == "websocket.connect":
await send_accept_task.wait()
disconnect_message = await receive()
response: typing.Optional[httpx.Response] = None
async def websocket_session(uri):
nonlocal response
async with httpx.AsyncClient() as client:
response = await client.get(
f"http://127.0.0.1:{unused_tcp_port}",
headers={
"upgrade": "websocket",
"connection": "upgrade",
"sec-websocket-version": "13",
"sec-websocket-key": "dGhlIHNhbXBsZSBub25jZQ==",
},
)
config = Config(
app=app,
ws=ws_protocol_cls,
http=http_protocol_cls,
lifespan="off",
port=unused_tcp_port,
)
async with run_server(config):
task = asyncio.create_task(
websocket_session(f"ws://127.0.0.1:{unused_tcp_port}")
)
await asyncio.sleep(0.1)
send_accept_task.set()
task.cancel()
> assert response is not None
E assert None is not None
tests/protocols/test_websocket.py:754: AssertionError
----------------------------- Captured stderr call -----------------------------
INFO: Started server process [2037]
INFO: Uvicorn running on http://127.0.0.1:33179 (Press CTRL+C to quit)
INFO: Shutting down
------------------------------ Captured log call -------------------------------
INFO uvicorn.error:server.py:76 Started server process [2037]
INFO uvicorn.error:server.py:219 Uvicorn running on http://127.0.0.1:33179 (Press CTRL+C to quit)
INFO uvicorn.error:server.py:265 Shutting down
================== 4 failed, 561 passed in 148.51s (0:02:28) ===================
Could we modify or remove this test?
I could use some help figuring this out.
This PR caused the flaky behavior: https://github.com/encode/uvicorn/pull/1804
Revert PR welcome, or solution to the problem... 😅
This PR caused the flaky behavior: #1804
Revert PR welcome, or solution to the problem... 😅
Thank you for pointing that out. I will take a look into it.
If possible, please consider covering the ssl_version parameter as well. Trying to update gunicorn to 21.x we started to see this error:
ai-api-1 | [2023-07-27 11:00:32 +0300] [766] [INFO] Booting worker with pid: 766
ai-api-1 | [2023-07-27 11:00:32 +0300] [760] [ERROR] Exception in worker process
ai-api-1 | Traceback (most recent call last):
ai-api-1 | File "/opt/venv/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
ai-api-1 | worker.init_process()
ai-api-1 | File "/opt/venv/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
ai-api-1 | super(UvicornWorker, self).init_process()
ai-api-1 | File "/opt/venv/lib/python3.10/site-packages/gunicorn/workers/base.py", line 142, in init_process
ai-api-1 | self.run()
ai-api-1 | File "/opt/venv/lib/python3.10/site-packages/uvicorn/workers.py", line 98, in run
ai-api-1 | return asyncio.run(self._serve())
ai-api-1 | File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
ai-api-1 | return loop.run_until_complete(main)
ai-api-1 | File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
ai-api-1 | File "/opt/venv/lib/python3.10/site-packages/uvicorn/workers.py", line 93, in _serve
ai-api-1 | await server.serve(sockets=self.sockets)
ai-api-1 | File "/opt/venv/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
ai-api-1 | config.load()
ai-api-1 | File "/opt/venv/lib/python3.10/site-packages/uvicorn/config.py", line 430, in load
ai-api-1 | self.ssl: Optional[ssl.SSLContext] = create_ssl_context(
ai-api-1 | File "/opt/venv/lib/python3.10/site-packages/uvicorn/config.py", line 119, in create_ssl_context
ai-api-1 | ctx = ssl.SSLContext(ssl_version)
ai-api-1 | File "/usr/local/lib/python3.10/ssl.py", line 496, in __new__
ai-api-1 | self = _SSLContext.__new__(cls, protocol)
ai-api-1 | TypeError: 'str' object cannot be interpreted as an integer
Relevant code:
# prestart.sh
export GUNICORN_CMD_ARGS=" \
--ssl-version=5 \
--cert-reqs=2 \
--ca-certs=/app/cert/[...] \
--keyfile=/app/cert/[...] \
--certfile=/app/cert/[...]"
# start.sh
# ... random stuff
# Start Gunicorn
exec python -m gunicorn -k "$WORKER_CLASS" -c "$GUNICORN_CONF" "$APP_MODULE"
I'm going to push this code to https://pypi.org/project/uvicorn-worker/ when I have a bit more time, and deprecate the uvicorn worker on uvicorn code source.
- This is merged on https://github.com/Kludex/uvicorn-worker/pull/5.
@br3ndonland I've invited you as a collaborator. I'll be closing this, and deprecating the worker from uvicorn.