"PROTOCOL_ERROR 0x1" when using hyper.HTTP20Adapter with python-requests

[pawelmhm]

I have simple Twisted resource looking like this

import sys

from twisted.web import server
from twisted.web.resource import Resource
from twisted.internet import reactor
from twisted.python import log

from twisted.internet import endpoints

class Index(Resource):
    isLeaf = True

    def render_GET(self, request):
        return b"hello world (in HTTP2)"


if __name__ == "__main__":
    log.startLogging(sys.stdout)
    site = server.Site(Index())
    endpoint_spec = "ssl:port=8080:privateKey=key.pem:extraCertChain=cert.pem"
    server = endpoints.serverFromString(reactor, endpoint_spec)
    server.listen(site)
    reactor.run()⏎    

and two hyper clients. One looks like this and works fine:

from hyper import HTTPConnection

conn = HTTPConnection('localhost:8080', secure=True)
conn.request('GET', '/')
resp = conn.get_response()
print(resp.read())

Another one does same thing but in a different way and fails

import requests
from hyper.contrib import HTTP20Adapter

s = requests.Session()
s.mount('https://localhost:8080', HTTP20Adapter())
r = s.get("https://localhost:8080")
print(r.status_code)

Traceback is:

Traceback (most recent call last):
  File "client2.py", line 6, in <module>
    r = s.get("https://localhost:8080")
  File "/home/pawel/.virtualenvs/http2/lib/python3.4/site-packages/requests/sessions.py", line 487, in get
    return self.request('GET', url, **kwargs)
  File "/home/pawel/.virtualenvs/http2/lib/python3.4/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/pawel/.virtualenvs/http2/lib/python3.4/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/home/pawel/.virtualenvs/http2/lib/python3.4/site-packages/hyper/contrib.py", line 80, in send
    resp = conn.get_response()
  File "/home/pawel/.virtualenvs/http2/lib/python3.4/site-packages/hyper/common/connection.py", line 129, in get_response
    return self._conn.get_response(*args, **kwargs)
  File "/home/pawel/.virtualenvs/http2/lib/python3.4/site-packages/hyper/http20/connection.py", line 293, in get_response
    return HTTP20Response(stream.getheaders(), stream)
  File "/home/pawel/.virtualenvs/http2/lib/python3.4/site-packages/hyper/http20/stream.py", line 223, in getheaders
    self._recv_cb(stream_id=self.stream_id)
  File "/home/pawel/.virtualenvs/http2/lib/python3.4/site-packages/hyper/http20/connection.py", line 744, in _recv_cb
    self._single_read()
  File "/home/pawel/.virtualenvs/http2/lib/python3.4/site-packages/hyper/http20/connection.py", line 701, in _single_read
    raise ConnectionError(error_string)
hyper.http20.exceptions.ConnectionError: Encountered error PROTOCOL_ERROR 0x1: Protocol error detected

why does it happen? Is my usage of adapter correct? Are both clients completely equivalent? If yes, why one works and other fails?

I'm using Python 3.4 on Ubuntu 14.04, OpenSSL 1.0.1.

[Lukasa]

With your adapter, can you try changing the s.get line from r = s.get("https://localhost:8080") to r = s.get("https://localhost:8080/")?

[pawelmhm]

With your adapter, can you try changing the s.get line from r = s.get("https://localhost:8080") to r = s.get("https://localhost:8080/")?

same behavior.

I actually see that example from docs also fails (albeit with different error).

# client2.py 
import requests
from hyper.contrib import HTTP20Adapter

s = requests.Session()

url = 'https://http2bin.org'
s.mount(url, HTTP20Adapter())
r = s.get("https://http2bin.org/get")
print(r.status_code)

and then I do something along the lines of :

> python --version
Python 2.7.6

> pip list
attrs (16.0.0)
backports.shutil-get-terminal-size (1.0.0)
cffi (1.7.0)
cryptography (1.4)
decorator (4.0.10)
enum34 (1.1.6)
h2 (2.4.0)
hpack (2.2.0)
hyper (0.6.2)
hyperframe (3.2.0)
idna (2.1)
ipaddress (1.0.16)
ipython (5.0.0)
ipython-genutils (0.1.0)
pathlib2 (2.1.0)
pexpect (4.2.0)
pickleshare (0.7.3)
pip (8.1.2)
prompt-toolkit (1.0.3)
ptyprocess (0.5.1)
pyasn1 (0.1.9)
pyasn1-modules (0.0.8)
pycparser (2.14)
Pygments (2.1.3)
pyOpenSSL (16.0.0)
requests (2.10.0)
service-identity (16.0.0)
setuptools (25.1.0)
simplegeneric (0.8.1)
six (1.10.0)
traitlets (4.2.2)
wcwidth (0.1.7)
wheel (0.29.0)

> python client2.py 
Traceback (most recent call last):
  File "client2.py", line 2, in <module>
    from hyper.contrib import HTTP20Adapter
  File "/home/pawel/.virtualenvs/hyp/local/lib/python2.7/site-packages/hyper/__init__.py", line 11, in <module>
    from .common.connection import HTTPConnection
  File "/home/pawel/.virtualenvs/hyp/local/lib/python2.7/site-packages/hyper/common/connection.py", line 10, in <module>
    from ..http20.connection import HTTP20Connection
  File "/home/pawel/.virtualenvs/hyp/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 35, in <module>
    TRANSIENT_SSL_ERRORS = (ssl.SSL_ERROR_WANT_READ, ssl.SSL_ERROR_WANT_WRITE)
AttributeError: 'module' object has no attribute 'SSL_ERROR_WANT_READ'

[Lukasa]

Yeah, that error occurs in the released version when using old Python versions. The master branch shouldn't have that issue.

So the protocol error is odd. Does the server generate any logs or output?

[vojd11]

same error with python3

<ipython-input-43-d09b7c0922c6> in <module>()
      3 s = requests.Session()
      4 s.mount('https://http2bin.org', HTTP20Adapter())
----> 5 r = s.get('https://http2bin.org/get')
      6 print(r.status_code)

/home/liza/.local/lib/python3.5/site-packages/requests/sessions.py in get(self, url, **kwargs)
    486 
    487         kwargs.setdefault('allow_redirects', True)
--> 488         return self.request('GET', url, **kwargs)
    489 
    490     def options(self, url, **kwargs):

/home/liza/.local/lib/python3.5/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    473         }
    474         send_kwargs.update(settings)
--> 475         resp = self.send(prep, **send_kwargs)
    476 
    477         return resp

/home/liza/.local/lib/python3.5/site-packages/requests/sessions.py in send(self, request, **kwargs)
    594 
    595         # Send the request
--> 596         r = adapter.send(request, **kwargs)
    597 
    598         # Total elapsed time of the request (approximately)

/usr/local/lib/python3.5/dist-packages/hyper/contrib.py in send(self, request, stream, cert, **kwargs)
     78             request.headers
     79         )
---> 80         resp = conn.get_response()
     81 
     82         r = self.build_response(request, resp)

/usr/local/lib/python3.5/dist-packages/hyper/common/connection.py in get_response(self, *args, **kwargs)
    127         """
    128         try:
--> 129             return self._conn.get_response(*args, **kwargs)
    130         except HTTPUpgrade as e:
    131             # We upgraded via the HTTP Upgrade mechanism. We can just

/usr/local/lib/python3.5/dist-packages/hyper/http20/connection.py in get_response(self, stream_id)
    291         """
    292         stream = self._get_stream(stream_id)
--> 293         return HTTP20Response(stream.getheaders(), stream)
    294 
    295     def get_pushes(self, stream_id=None, capture_all=False):

/usr/local/lib/python3.5/dist-packages/hyper/http20/stream.py in getheaders(self)
    221         # Keep reading until all headers are received.
    222         while self.response_headers is None:
--> 223             self._recv_cb(stream_id=self.stream_id)
    224 
    225         # Find the Content-Length header if present.

/usr/local/lib/python3.5/dist-packages/hyper/http20/connection.py in _recv_cb(self, stream_id)
    742 
    743             # TODO: Re-evaluate this.
--> 744             self._single_read()
    745             count = 9
    746             retry_wait = 0.05  # can improve responsiveness to delay the retry

/usr/local/lib/python3.5/dist-packages/hyper/http20/connection.py in _single_read(self)
    699                         )
    700 
--> 701                     raise ConnectionError(error_string)
    702             else:
    703                 log.info("Received unhandled event %s", event)

ConnectionError: Encountered error PROTOCOL_ERROR 0x1: Protocol error detected

[Lukasa]

Can you please print out the result of pip freeze on that system?

[ZeeMuFaSaa]

Having the same issue when the doc example code as pawelmhm above. Here is a 'pip freeze' output dump. Using python 3.4.3 in ubuntu 14.04.

Brlapi==0.6.1 CherryPy==8.1.0 Magic-file-extensions==0.2 Mako==0.9.1 MarkupSafe==0.18 PyAudio==0.2.9 apturl===0.5.2ubuntu4 chardet==2.2.1 checkbox-ng==0.3 checkbox-support==0.2 colorama==0.2.5 command-not-found==0.3 decorator==4.0.9 defer==1.0.6 devscripts===2.14.1ubuntu0.1 dict==0.0.5 feedparser==5.1.3 friends==0.1 get==0.0.7 h2==2.4.1 hpack==2.3.0 html5lib==0.999 httplib2==0.8 hyper==0.6.2 hyperframe==3.2.0 language-selector==0.1 louis==2.5.3 lxml==3.3.3 numpy==1.11.1 oauthlib==0.6.1 onboard==1.0.1 oneconf==0.3.7.14.4.1 piston-mini-client==0.7.5 plainbox==0.5.3 pocketsphinx==0.1.3 post==0.0.5 public==0.0.0 pyOpenSSL==16.1.0 pycrypto==2.6.1 pycurl==7.19.3 pygobject==3.12.0 pyparsing==2.0.1 pyqtgraph==0.9.10 python-apt===0.9.3.5ubuntu2 python-debian===0.1.21-nmu2ubuntu2 pyxdg==0.25 query-string==0.0.5 request==0.0.5 requests==2.10.0 scipy==0.13.3 self==0.0.5 six==1.5.2 software-center-aptd-plugins==0.0.0 ubuntu-drivers-common==0.0.0 ufw===0.34-rc-0ubuntu2 unattended-upgrades==0.1 unity-scope-audacious==0.1 unity-scope-calculator==0.1 unity-scope-chromiumbookmarks==0.1 unity-scope-clementine==0.1 unity-scope-colourlovers==0.1 unity-scope-devhelp==0.1 unity-scope-firefoxbookmarks==0.1 unity-scope-gdrive==0.7 unity-scope-gmusicbrowser==0.1 unity-scope-gourmet==0.1 unity-scope-guayadeque==0.1 unity-scope-manpages==0.1 unity-scope-musique==0.1 unity-scope-openclipart==0.1 unity-scope-texdoc==0.1 unity-scope-tomboy==0.1 unity-scope-virtualbox==0.1 unity-scope-yelp==0.1 unity-scope-zotero==0.1 urllib3==1.7.1 usb-creator==0.2.23 wheel==0.24.0 xdiagnose===3.6.3build2 xkit==0.0.0

[ZeeMuFaSaa]

Additionally, consistently get the following error when trying another example from the doc:

Python 3.4.3 (default, Sep 14 2016, 12:36:27) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information.

from hyper import HTTPConnection c = HTTPConnection('http2bin.org') first = c.request('GET', '/get', headers={'key': 'value'}) second = c.request('POST', '/post', body=b'hello') third = c.request('GET', '/ip') second_response = c.get_response(second) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.4/dist-packages/hyper/common/connection.py", line 129, in get_response return self._conn.get_response(_args, *_kwargs) TypeError: get_response() takes 1 positional argument but 2 were given first_response = c.get_response(first) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.4/dist-packages/hyper/common/connection.py", line 129, in get_response return self._conn.get_response(_args, *_kwargs) TypeError: get_response() takes 1 positional argument but 2 were given

However, if I use the HTTP20Connection class, then all works as advertised. It would appear that the HTTPConnection class is not connecting utilizing HTTP/2. The same happens in other examples in the doc. I assumed that the HTTPConnection would automatically recognize a HTTP/2 site(Upgade). Is this a documentation issue, or a bug?

Thanks

Greg

[Lukasa]

I assumed that the HTTPConnection would automatically recognize a HTTP/2 site(Upgade). Is this a documentation issue, or a bug?

It should. And indeed, on a quick test shows that it does. The error you're talking about in the most recent comment seems to be fundamentally a result of the fact that we are not reading the response until asked the first time, which means that we don't yet know that we're doing HTTP/2 instead of HTTP/1.1. That example works with port 443, but does not work with port 80.

It seems fundamentally to be unrelated to the protocol error discussed elsewhere in this issue, so I'm going to open a new issue for this problem (which involves in the first instance changing the documentation).