cheroot
cheroot copied to clipboard
Published
20 hours ago •
cherrypy
cherrypy
Make HTTP server correctly read trailers (Trailer HTTP headers, which come after body)
- I'm submitting a ...
- [X] bug report
- [X] feature request
- [ ] question about the decisions made in the repository
-
Do you want to request a feature or report a bug? It's a bug I noticed during tests rewrite for pytest runner #67
-
What is the current behavior? Unread ending of the request with headers gets read by the next request parser, which recognises it as garbage (because it's not a valid Request-Line).
Extracted from the xfail message (removed in f55e41dd):
Server does not correctly read trailers/ending of the previous HTTP request, thus the second request fails as the server tries to parse b'Content-Type: application/json\r\n' as a Request-Line. This results in HTTP status code 400, instead of 413.
-
If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem. If you can, show us your code. See
conn_test.py::test_Chunked_Encoding. -
What is the expected behavior? Payload of the request with trailers is completely read out from the socket, so that it won't affect the next request.
-
What is the motivation / use case for changing the behavior? Emm.. My perfectionism, I guess.
-
Please tell us about your environment:
- Python version: 3.6.2 (but shouldn't version-agnostic)
- Cheroot version: v6.0.0 (and probably all prev versions)
- CherryPy version: N/A
- OS: Gentoo Linux
- Browser: N/A
- Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, e.g. stackoverflow, gitter, etc.)
Here's the test failure:
cheroot master $ tox -- --runxfail -k test_Chunked_Encoding
python develop-inst-noop: /Users/jaraco/code/public/cherrypy/cheroot
python installed: apipkg==1.5,argh==0.26.2,atomicwrites==1.3.0,attrs==19.3.0,certifi==2019.9.11,cffi==1.13.0,chardet==3.0.4,-e git+gh://cherrypy/cheroot@f55e41dd041392ec8504547f18dd5b8e838b5092#egg=cheroot,codecov==2.0.15,colorama==0.4.1,coverage==4.5.3,cryptography==2.8,docopt==0.6.2,execnet==1.7.1,idna==2.8,jaraco.functools==2.0,more-itertools==7.2.0,packaging==19.2,pathtools==0.1.2,pluggy==0.13.0,py==1.8.0,pycparser==2.19,pyOpenSSL==19.0.0,pyparsing==2.4.2,pytest==5.2.1,pytest-cov==2.7.1,pytest-forked==1.1.1,pytest-mock==1.11.1,pytest-sugar==0.9.2,pytest-testmon==0.9.19,pytest-watch==4.2.0,pytest-xdist==1.30.0,PyYAML==5.1.2,requests==2.22.0,requests-unixsocket==0.2.0,six==1.12.0,termcolor==1.1.0,trustme==0.5.2,urllib3==1.25.6,watchdog==0.9.0,wcwidth==0.1.7
python run-test-pre: PYTHONHASHSEED='4070659726'
python run-test: commands[0] | pytest --testmon-off --runxfail -k test_Chunked_Encoding
Test session starts (platform: darwin, Python 3.8.0, pytest 5.2.1, pytest-sugar 0.9.2)
cachedir: .tox/python/.pytest_cache
rootdir: /Users/jaraco/code/public/cherrypy/cheroot, inifile: pytest.ini, testpaths: cheroot/test/
plugins: testmon-0.9.19, xdist-1.30.0, forked-1.1.1, sugar-0.9.2, cov-2.7.1, mock-1.11.1
[gw0] darwin Python 3.8.0 cwd: /Users/jaraco/code/public/cherrypy/cheroot
[gw1] darwin Python 3.8.0 cwd: /Users/jaraco/code/public/cherrypy/cheroot
[gw2] darwin Python 3.8.0 cwd: /Users/jaraco/code/public/cherrypy/cheroot
[gw3] darwin Python 3.8.0 cwd: /Users/jaraco/code/public/cherrypy/cheroot
[gw4] darwin Python 3.8.0 cwd: /Users/jaraco/code/public/cherrypy/cheroot
[gw5] darwin Python 3.8.0 cwd: /Users/jaraco/code/public/cherrypy/cheroot
[gw6] darwin Python 3.8.0 cwd: /Users/jaraco/code/public/cherrypy/cheroot
[gw7] darwin Python 3.8.0 cwd: /Users/jaraco/code/public/cherrypy/cheroot
[gw0] Python 3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) -- [Clang 6.0 (clang-600.0.57)]
[gw1] Python 3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) -- [Clang 6.0 (clang-600.0.57)]
[gw2] Python 3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) -- [Clang 6.0 (clang-600.0.57)]
[gw3] Python 3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) -- [Clang 6.0 (clang-600.0.57)]
[gw4] Python 3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) -- [Clang 6.0 (clang-600.0.57)]
[gw5] Python 3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) -- [Clang 6.0 (clang-600.0.57)]
[gw6] Python 3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) -- [Clang 6.0 (clang-600.0.57)]
[gw7] Python 3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) -- [Clang 6.0 (clang-600.0.57)]
gw0 [1] / gw1 [1] / gw2 [1] / gw3 [1] / gw4 [1] / gw5 [1] / gw6 [1] / gw7 [1]
scheduling tests via LoadScheduling
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_Chunked_Encoding ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
[gw0] darwin -- Python 3.8.0 /Users/jaraco/code/public/cherrypy/cheroot/.tox/python/bin/python
test_client = <cheroot.testing._TestClient object at 0x10f934cd0>
@pytest.mark.xfail(
reason='https://github.com/cherrypy/cheroot/issues/69',
)
def test_Chunked_Encoding(test_client):
"""Test HTTP uploads with chunked transfer-encoding."""
# Initialize a persistent HTTP connection
conn = test_client.get_connection()
# Try a normal chunked request (with extensions)
body = (
b'8;key=value\r\nxx\r\nxxxx\r\n5\r\nyyyyy\r\n0\r\n'
b'Content-Type: application/json\r\n'
b'\r\n'
)
conn.putrequest('POST', '/upload', skip_host=True)
conn.putheader('Host', conn.host)
conn.putheader('Transfer-Encoding', 'chunked')
conn.putheader('Trailer', 'Content-Type')
# Note that this is somewhat malformed:
# we shouldn't be sending Content-Length.
# RFC 2616 says the server should ignore it.
conn.putheader('Content-Length', '3')
conn.endheaders()
conn.send(body)
response = conn.getresponse()
status_line, actual_headers, actual_resp_body = webtest.shb(response)
actual_status = int(status_line[:3])
assert actual_status == 200
assert status_line[4:] == 'OK'
expected_resp_body = ("thanks for '%s'" % b'xx\r\nxxxxyyyyy').encode()
assert actual_resp_body == expected_resp_body
# Try a chunked request that exceeds server.max_request_body_size.
# Note that the delimiters and trailer are included.
body = b'3e3\r\n' + (b'x' * 995) + b'\r\n0\r\n\r\n'
conn.putrequest('POST', '/upload', skip_host=True)
conn.putheader('Host', conn.host)
conn.putheader('Transfer-Encoding', 'chunked')
conn.putheader('Content-Type', 'text/plain')
# Chunked requests don't need a content-length
# conn.putheader("Content-Length", len(body))
conn.endheaders()
conn.send(body)
response = conn.getresponse()
status_line, actual_headers, actual_resp_body = webtest.shb(response)
actual_status = int(status_line[:3])
> assert actual_status == 413
E assert 400 == 413
E -400
E +413
actual_headers = [('Content-Length', '22'), ('Content-Type', 'text/plain')]
actual_resp_body = b'Malformed Request-Line'
actual_status = 400
body = (b'3e3\r\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
b'xxxxxxxxxxxxxxxx\r\n0\r\n\r\n')
conn = <http.client.HTTPConnection object at 0x10f934dc0>
expected_resp_body = b"thanks for 'b'xx\\r\\nxxxxyyyyy''"
response = <http.client.HTTPResponse object at 0x10f9348b0>
status_line = '400 Bad Request'
test_client = <cheroot.testing._TestClient object at 0x10f934cd0>
cheroot/test/test_conn.py:848: AssertionError
cheroot/test/test_conn.py::test_Chunked_Encoding ⨯ 100% ██████████
[gw0] FAILED cheroot/test/test_conn.py Coverage.py warning: No data was collected. (no-data-collected)
Exception ignored in: <socket.socket fd=14, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('::1', 51715, 0, 0)>
ResourceWarning: unclosed <socket.socket fd=14, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('::1', 51715, 0, 0)>
----------------------------------------- generated xml file: /Users/jaraco/code/public/cherrypy/cheroot/junit-test-results.xml ------------------------------------------
---------- coverage: platform darwin, python 3.8.0-final-0 -----------
Name Stmts Miss Cover Missing
-------------------------------------------------------------
cheroot/__init__.py 10 4 60% 8-9, 14-15
cheroot/__main__.py 3 3 0% 3-6
cheroot/_compat.py 49 21 57% 15-16, 40-42, 49-77, 88, 99, 104-110
cheroot/cli.py 71 71 0% 24-234
cheroot/connections.py 143 70 51% 19-41, 92, 97-98, 102-103, 128, 146-163, 168-170, 176, 182-183, 189, 198-225, 237-242, 249-273, 278
cheroot/makefile.py 302 241 20% 11-13, 32, 46-47, 56-59, 62, 65-68, 72-82, 86-88, 92-95, 100-110, 116, 132-190, 194-282, 286-406, 447
cheroot/server.py 938 434 54% 80-81, 122-129, 134-137, 197, 203, 209, 213, 217-218, 224, 229, 246-247, 266, 278-281, 294-297, 322-331, 335, 339, 343-346, 362-363, 375-384, 396-405, 418-427, 431, 435, 439-441, 471, 477, 486-487, 496, 504, 522, 526, 535-537, 552-583, 596-605, 614-635, 639, 699, 721-727, 734-740, 743, 774-776, 779-782, 787-790, 793-796, 799-802, 811-818, 824, 826-828, 834, 842-874, 885-889, 900-906, 909-913, 917-921, 938-940, 944, 947, 970-971, 983-985, 991-996, 999-1004, 1010-1014, 1032-1034, 1057-1063, 1072-1081, 1087, 1103-1112, 1122-1124, 1135-1137, 1151, 1156-1169, 1175-1179, 1194-1196, 1272, 1283-1311, 1316-1330, 1338-1346, 1376-1408, 1413-1414, 1419-1420, 1425-1426, 1436-1450, 1455-1456, 1461-1462, 1476, 1632-1635, 1639, 1676, 1687-1696, 1714, 1717-1721, 1732-1740, 1747-1751, 1754, 1771-1774, 1780-1784, 1807-1812, 1827-1908, 1930, 1938, 1953-1956, 1983-1986, 1991, 1997-1999, 2011-2013, 2031-2036, 2056-2058, 2093-2114
cheroot/ssl/__init__.py 22 22 0% 3-50
cheroot/ssl/builtin.py 90 90 0% 10-210
cheroot/ssl/pyopenssl.py 140 140 0% 34-343
cheroot/test/conftest.py 38 22 42% 31, 37-54, 59-69
cheroot/test/helper.py 102 47 54% 48-76, 81-82, 87-89, 94-97, 103-111, 135, 138-141, 154-155, 164-166
cheroot/test/test__compat.py 24 10 58% 23, 35-37, 42-44, 56, 61-62
cheroot/test/test_conn.py 517 422 18% 26, 30, 34-41, 46, 52-53, 57-58, 62-63, 67, 71-72, 76-77, 85-87, 112, 131, 136, 145-182, 195-261, 273-334, 346-390, 395-465, 480-497, 505-591, 599-634, 639-689, 701-761, 767-799, 849, 859-875, 880-890, 914-926, 936-954, 967-976
cheroot/test/test_core.py 205 147 28% 30, 34-37, 41, 45-47, 56, 72-76, 82-86, 92, 97-99, 104-107, 112-117, 132-134, 153-164, 172-179, 194-199, 207-211, 216-220, 225-231, 241-247, 256-262, 291-298, 303-312, 317-328, 333-345, 352-366, 374-377, 381, 389-391, 395, 399, 405-409, 414-415
cheroot/test/test_dispatch.py 15 11 27% 12-24, 30-49
cheroot/test/test_errors.py 8 3 62% 28-30
cheroot/test/test_makefile.py 31 22 29% 14, 18-23, 27-30, 34, 39-43, 48-52
cheroot/test/test_server.py 126 83 34% 45-53, 59-64, 72-90, 95-110, 122-125, 131-133, 139-141, 150-160, 163-166, 172-175, 182-201, 213-235
cheroot/test/test_ssl.py 178 125 30% 79-86, 91-105, 111, 117, 123-124, 130-131, 137-138, 144-145, 163-188, 227-354, 366-379, 406-474
cheroot/test/webtest.py 323 235 27% 46, 50, 56-59, 64-66, 74-81, 96-98, 127-128, 132-136, 147-152, 161, 165, 173, 198-228, 238-246, 251-296, 301, 305-310, 318-323, 327-338, 342-351, 355-362, 366-371, 375-381, 385-392, 396-401, 405-410, 414-419, 427-456, 466-481, 493-503, 517-546, 567-570, 596-605
cheroot/testing.py 78 17 78% 48-49, 74-75, 102, 111-115, 124-128, 132, 139, 144-146
cheroot/workers/threadpool.py 152 56 63% 25, 28, 116-117, 122, 132-136, 138-139, 183, 188-189, 202-207, 211-222, 225-228, 234-250, 261-262, 291-292, 297-308, 323
cheroot/wsgi.py 162 67 59% 92, 96, 147, 153, 160, 170-173, 179, 183, 203, 205, 215, 220-230, 239, 282-292, 310, 316, 332-349, 353-356, 360-364, 380-392, 409-424
-------------------------------------------------------------
TOTAL 3746 2363 37%
3 files skipped due to complete coverage.
Coverage XML written to file coverage.xml
======================================================================= slowest 10 test durations ========================================================================
0.11s setup cheroot/test/test_conn.py::test_Chunked_Encoding
0.11s call cheroot/test/test_conn.py::test_Chunked_Encoding
(0.00 durations hidden. Use -vv to show these durations.)
Results (2.86s):
1 failed
- cheroot/test/test_conn.py:802 test_Chunked_Encoding
ERROR: InvocationError for command /Users/jaraco/code/public/cherrypy/cheroot/.tox/python/bin/pytest --testmon-off --runxfail -k test_Chunked_Encoding (exited with code 1)
________________________________________________________________________________ summary _________________________________________________________________________________
ERROR: python: commands failed
