aiohttp
aiohttp copied to clipboard
aio-libs
Server throws 'aiohttp.http_exceptions.BadStatusLine'
Long story short
Not sure if this is a user error, or a bug.
On a bad request, the server raise aiohttp.http_exceptions.BadStatusLine: invalid HTTP method, and custom headers are ignored. I can't find any way to catch this exception.
My application works as it should, and custom headers are sent on both GET and POST, and also HEAD - using the on_prepare signal.
I want to use custom headers and hide the Server: Python/3.6 aiohttp/3.4.4. But a simple request containing only "/" bypass all handlers, and display the default headers. And also raise a seemingly non-handable exception.
Expected behaviour
I'd expect being able to handle all requests, no matter how erranous.
Actual behaviour
In console, AioHttp drops
HTTP/1.0 400 Bad Request
Content-Type: text/html; charset=utf-8
Content-Length: 11
Date: Mon, 24 Sep 2018 14:13:37 GMT
Server: Python/3.6 aiohttp/3.4.4
Bad Request
and in logs:
Traceback (most recent call last):
File "site-packages\aiohttp\web_protocol.py", line 242, in data_received
File "aiohttp\_http_parser.pyx", line 523, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: invalid HTTP method
Steps to reproduce
- Start a barebone server (like in Getting started)
- telnet 127.0.0.1 8080, curl should work as well
- type "/"
Your environment
Windows Python/3.6 aiohttp/3.4.4 aiohttp server
GitMate.io thinks possibly related issues are https://github.com/aio-libs/aiohttp/issues/2632 (aiohttp.http_exceptions.BadStatusLine: invalid HTTP method), https://github.com/aio-libs/aiohttp/issues/1619 (aiohttp.Client can't parse Trailer headers and throws exception), https://github.com/aio-libs/aiohttp/issues/1792 (aiohttp.http_exceptions.BadStatusLine caused by extra CRLF after POST data), https://github.com/aio-libs/aiohttp/issues/58 (aiohttp.HttpClient), and https://github.com/aio-libs/aiohttp/issues/1390 (Build chat server with aiohttp).
The best we can do here is dropping connection without any output (but logging the fact).
BadStatusLine is very exceptional.
In general, it is a kind of parser error. Nothing to do here, no user code to call.
No, server should reply with 400 Bad Request.
Yes, different servers act differently:
➜ telnet google.com 80
Trying 2a00:1450:4014:801::200e...
telnet: connect to address 2a00:1450:4014:801::200e: No route to host
Trying 172.217.23.206...
Connected to google.com.
Escape character is '^]'.
/
HTTP/1.0 400 Bad Request
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1555
Date: Thu, 27 Sep 2018 08:27:56 GMT
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 400 (Bad Request)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>400.</b> <ins>That’s an error.</ins>
<p>Your client has issued a malformed or illegal request. <ins>That’s all we know.</ins>
Connection closed by foreign host.
➜ telnet twitter.com 80
Trying 104.244.42.193...
Connected to twitter.com.
Escape character is '^]'.
/
Connection closed by foreign host.
However, let's clarify terms. The first line is called Request Line in HTTP request and Status Line in HTTP response. So BadStatusLine is a bit misleading name for the exception.
Now, if you look at RFC 7230, section 3.1.2, it clearly recommended to send a reply, not drop the connection immediately:
Recipients of an invalid request-line SHOULD respond with either a 400 (Bad Request) error or a 301 (Moved Permanently) redirect with the request-target properly encoded. A recipient SHOULD NOT attempt to autocorrect and then process the request without a redirect, since the invalid request-line might be deliberately crafted to bypass security filters along the request chain.
So if we want to follow RFCs, we have to do this and not reset a TCP connection. For some users, though, who need this connection resetting may be provided via feature flag or recommendations of how to do this manually.
@asvetlov the topic starter wants to define some HTTP handlers probably for consistency with valid requests (I'd not want to expose things to attacker who tries to find out which web server I use with just sending a malicious request).
Oh, and just as an idea: in CherryPy we have a setting/hook for error codes, which specifies what to send in payload. It can be either a string or a callable.
Maybe you could think of some similar concept within aiohttp architecture or add a recommendation to docs on how to process such things.
In aiohttp, we use middlewares for error handling. The fix should follow this way I guess (but still support low-level HTTP server)
So the problem is that currently you can't catch this with middleware, right?
The thread starts from asking for handling the error by on_prepare signal.
Middleware is another way to handle it.
Hi. Sorry for late reply and thanks for the discussion. For my application, this bug has the effect of allowing a client to bypass any attempts to serve custom headers passed in a WebResponse or by web.Application.on_response_prepare.append(). Else, the bug is more cosmetic, or principal, depending on how it's considered.
I'd prefer AioHttp to be RFC compliant, and not being able to handle any exception that's raised, seems weird. Perhaps catching it in system code/low-level code, and raise aiohttp.web.HTTPBadRequest is a viable way.
More seasoned coders than me should fix this. I would't know the wider effects of any change I'd make.
A bit confused - is this the same exception as in Standard lib's http.client.BadStatusLine, derived from HTTPException?
Thanks for all the work, and a very nice product!
No, aiohttp.web_exceptions are not derived from http.client ones.
There are many reasons for it but the first is: server-side exceptions for 400+ status codes have no common meaning with 400+ statuses received by client.
Update - This behaviour disappeared when adding a ssl_context to the web.TCPSite().
Heh, encrypted SSL channel doesn't look like a regular plain HTTP status line, isn't it?
Can aiohttp include any more details of the request in the exception? I'm encountering this issue in my tests after upgrading from aiohttp 3.3.2 to 3.5.4, and I have no idea what's causing it.
So the problem is that currently you can't catch this with middleware, right?
That's certainly my problem, signals are no help either. Not only can't I add extra headers to the response, but I also can't suppress these errors which aren't useful to me since I cannot do anything about them.
[Problem certainly is there with SSL as well (unsurprisingly), so long as you actually establish the SSL connection (i.e. openssl s_client -connect
There is no well-formed request exists, no middleware is affected. The error happens on the HTTP protocol level, even before aiohttp routing starts to work. In other words, middlewares/signals are not called if the HTTP protocol is broken. There is no peer to send a response at all etc.
The only thing we can do is the conversion of exception with the attached stacktrace into a logged warning. A volunteer is welcome!
You're correct that there's no well-formed request, and I don't think calling middlewares makes any sense either, however something catches the error and responds to the client with a well-formed 400 response.
A volunteer is welcome!
What do you think is the right thing to do though?
Removing the stacktrace and adding the peer's IP address to the error message (logged to aiohttp.server) would be nice (and trivial, I imagine), but that doesn't help with what the OP wanted (and what brought me here): customizing headers. For this, it's tempting to reuse signals but calling them with either "None" or a minimalist request seems likely to cause problems. Another option could be to allow customizing the default error pages, beyond what https://docs.aiohttp.org/en/stable/web_advanced.html#handling-error-pages suggests (which doesn't apply here).
The only thing we can do is the conversion of exception with the attached stacktrace into a logged warning.
@asvetlov can you add more details what exactly should happen in this case?
you can try this to block the log: app = web.Application() app.logger.manager.disable = 100
@rodusluo where did you find this?
If anyone reading this was able to provide explanation on how to validate a HTTP request before it hits AIOHTTP that would probably go someway to solving the issue here.
before it hits AIOHTTP that would probably go someway to solving the issue here.
Put a reverse proxy in front?