Problem sending JSON data as request body

[lily-mara]

I am trying to do some testing using the http2 version of httbin: https://nghttp2.org/httpbin/. Posting to the url https://nghttp2.org/httpbin/post will respond with a JSON object containing information about the request. This JSON includes the body and data sent with the request. Using hyper, it appears that the data that I'm sending as the request body is not being sent. Here's an example:

from hyper import HTTPConnection
conn = HTTPConnection('nghttp2.org:443')

req = conn.request(
    'POST',
    '/httpbin/post',
    body=b'{"foo": "bar"}',
)

resp = conn.get_response()
print(resp.read())

Using a different http2 client, I get the expected response:

{
  "args": {},
  "data": "{\"foo\": \"bar\"}",  <-- critical bit right here
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Accept-Language": "en-US,en;q=0.8",
    "Cache-Control": "no-cache",
    "Content-Length": "14",
    "Content-Type": "application/json",
    "Csp": "active",
    "Host": "nghttp2.org",
    "Origin": "chrome-extension://aicmkgpgakddgnaphhhpliifpcfhicfo",
    "Postman-Token": "70fa24f0-9e35-82ef-c838-7c15461ba3bd",
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/43.0.2357.130 Chrome/43.0.2357.130 Safari/537.36",
    "Via": "2 nghttpx"
  },
  "json": {
    "foo": "bar"
  },
  "origin": "68.180.88.224",
  "url": "https://nghttp2.org/httpbin/post"
}

But using the python given above, I get the response:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Host": "nghttp2.org",
    "Transfer-Encoding": "chunked",
    "Via": "2 nghttpx"
  },
  "json": null,
  "origin": "68.180.88.224",
  "url": "https://nghttp2.org/httpbin/post"
}

I'm using version 0.5.0 of hyper, Python 3.4.4. Am I doing something wrong with my request, or is there a way to get more verbose output from hyper so that I can see what's really going on?

[Lukasa]

@natemara Can you quickly test whether this problem is fixed by using the code from the master branch on GitHub?

[lily-mara]

I installed directly from master, and I'm getting the same results.

[Lukasa]

Interesting. So this seems only to happen with nghttp2.org: with http2bin.org, the same problem is not observed.

@tatsuhiro-t, do you have any theories about why nghttp2.org/httpbin might not be receiving data?

[tatsuhiro-t]

The difference between working client (e.g., nghttp) and hyper client is that former sends content-length, but hyper does not. nghttpx (which forwards request to backend httpbin) uses Transfer-Encoding: chunked in HTTP/1, but httpbin seems to not support chunked request. I have not verified httpbin code yet. I just checked that request body without content-length was sent to backend using go HTTP server as nghttpx's backend. So probably, httpbin does not handle request body without content-length yet?

[Lukasa]

@tatsuhiro-t That's interesting: what WSGI server are you using to run httpbin?

[tatsuhiro-t]

Gunicorn. Does it answer your question?

[Lukasa]

@tatsuhiro-t Hmm, that's interesting. http2bin.org uses H2O and gunicorn and doesn't display this problem. Weird.

[tatsuhiro-t]

Will check how h2o behaves in this situation. I think it is possible to calculate content-length if all request body is buffered. BTW, why doesn't hyper client set content-length in request? Certainly, there is no way to set, but in this case, it knows whole request body, so it can set content-length.

[Lukasa]

@tatsuhiro-t Yeah, I think the lack of Content-Length in this case in hyper is a bug. That's ok. =)

[tatsuhiro-t]

I checked what h2o does. As I guessed, h2o sets content-length if the request body fits in its buffer.

[Lukasa]

Good intuition! =)