tavern icon indicating copy to clipboard operation
tavern copied to clipboard
verified publisher icon taverntesting

Unable to use imports with stages (see components example)

Open hamx0r opened this issue 4 years ago • 6 comments

After running the Dockerized flask server in the "components example" (https://github.com/taverntesting/tavern/tree/master/example/components) , I then run the py.test tests in the same directory. The test fails with the below, which is a similar failure I'm experiencing in my own code project (I'm new to Tavern). If I put variables in a common file, I can import and use those variables. However, if I have a common stages section I want run (ie login and save the JWT token), it appears successive tests can't make use of such an import. Such is the case in the example.

/Users/hamx0r/PyCharmProjects/tavern/venv/bin/python /Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/_jb_pytest_runner.py --path /Users/hamx0r/PyCharmProjects/tavern/example/components
Testing started at 4:31 PM ...
Launching pytest with arguments /Users/hamx0r/PyCharmProjects/tavern/example/components --no-header --no-summary -q in /Users/hamx0r/PyCharmProjects/tavern/example/components

============================= test session starts ==============================
collecting ... collected 2 items

test_hello.tavern.yaml::Test authenticated /hello FAILED                 [ 50%]Error running prepared request
Traceback (most recent call last):
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/Users/hamx0r/.pyenv/versions/3.6.14/lib/python3.6/http/client.py", line 1291, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/hamx0r/.pyenv/versions/3.6.14/lib/python3.6/http/client.py", line 1337, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/hamx0r/.pyenv/versions/3.6.14/lib/python3.6/http/client.py", line 1286, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/hamx0r/.pyenv/versions/3.6.14/lib/python3.6/http/client.py", line 1046, in _send_output
    self.send(msg)
  File "/Users/hamx0r/.pyenv/versions/3.6.14/lib/python3.6/http/client.py", line 984, in send
    self.connect()
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x1111e0780>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5009): Max retries exceeded with url: /hello/Jim (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1111e0780>: Failed to establish a new connection: [Errno 61] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/hamx0r/PyCharmProjects/tavern/tavern/_plugins/rest/request.py", line 483, in run
    return self._prepared()
  File "/Users/hamx0r/PyCharmProjects/tavern/tavern/_plugins/rest/request.py", line 463, in prepared_request
    return session.request(**self._request_args)
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5009): Max retries exceeded with url: /hello/Jim (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1111e0780>: Failed to establish a new connection: [Errno 61] Connection refused',))

example/components/test_hello.tavern.yaml:0 (/Users/hamx0r/PyCharmProjects/tavern/example/components/test_hello.tavern.yaml::Test authenticated /hello)
Format variables:
  service:s = 'http://localhost:5009'

Source test stage (line 9):
  - name: Unauthenticated /hello
    request:
      url: "{service:s}/hello/Jim"
      method: GET
    response:
      status_code: 401

Formatted stage:
  name: Unauthenticated /hello
  request:
    method: GET
    url: 'http://localhost:5009/hello/Jim'
  response:
    status_code: 401

Errors:
E   tavern.util.exceptions.RestRequestException




FAILED                   [100%]Error running prepared request
Traceback (most recent call last):
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/Users/hamx0r/.pyenv/versions/3.6.14/lib/python3.6/http/client.py", line 1291, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/hamx0r/.pyenv/versions/3.6.14/lib/python3.6/http/client.py", line 1337, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/hamx0r/.pyenv/versions/3.6.14/lib/python3.6/http/client.py", line 1286, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/hamx0r/.pyenv/versions/3.6.14/lib/python3.6/http/client.py", line 1046, in _send_output
    self.send(msg)
  File "/Users/hamx0r/.pyenv/versions/3.6.14/lib/python3.6/http/client.py", line 984, in send
    self.connect()
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x110d95160>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5009): Max retries exceeded with url: /ping (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x110d95160>: Failed to establish a new connection: [Errno 61] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/hamx0r/PyCharmProjects/tavern/tavern/_plugins/rest/request.py", line 483, in run
    return self._prepared()
  File "/Users/hamx0r/PyCharmProjects/tavern/tavern/_plugins/rest/request.py", line 463, in prepared_request
    return session.request(**self._request_args)
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/hamx0r/PyCharmProjects/tavern/venv/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5009): Max retries exceeded with url: /ping (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x110d95160>: Failed to establish a new connection: [Errno 61] Connection refused',))

example/components/test_ping.tavern.yaml:0 (/Users/hamx0r/PyCharmProjects/tavern/example/components/test_ping.tavern.yaml::Test authenticated /ping)
Format variables:
  service:s = 'http://localhost:5009'

Source test stage (line 9):
  - name: Unauthenticated /ping
    request:
      url: "{service:s}/ping"
      method: GET
    response:
      status_code: 401

Formatted stage:
  name: Unauthenticated /ping
  request:
    method: GET
    url: 'http://localhost:5009/ping'
  response:
    status_code: 401

Errors:
E   tavern.util.exceptions.RestRequestException








test_ping.tavern.yaml::Test authenticated /ping 

============================== 2 failed in 1.00s ===============================

Process finished with exit code 1

hamx0r avatar Nov 24 '21 00:11 hamx0r

hi, @hamx0r

Judging from the error message, it is that http://localhost:5009 is not yet ready, Is it normal for you to visit http://localhost:5009 by browser？ Can your additional statement how do you start docker?

dongfangtianyu avatar Nov 24 '21 08:11 dongfangtianyu

Thanks for your quick reply. To reproduce the above, I think one needs to do this:

  1. Pull this project's code (especially the code in this path: https://github.com/taverntesting/tavern/blob/master/example/components/ )
  2. cd example/components
  3. Run docker-compose up . This creates the test server listening on port 5009 which you noticed.
  4. Now, run the Tavern tests for this feature: py.test in the same local directory
  5. Observe errors above.

One can verify the test server is running by visiting http://localhost:5009/ping in a browser. (returns {"error":"No token"}). One can even try to login by imitating what auth_stage.yaml does curl -X POST --location "http://localhost:5009/login" -H "Content-Type: application/json" -d "{\"user\": \"test-user\", \"password\": \"correct-password\"}"

This fails in the same way.

Looking at the Docker logs for the server, we get 2 useful clues:

  1. the JSON body is not being parsed correctly
  2. The Tavern tests are not run "in order". I'd expect auth_parse.yaml to get run first since it's an import. However, we see the GET request from test_hello.tavern.yaml getting run first (and failing with 401 since not authenticated) Perhaps there are 2 errors present here instead of just one.
172.19.0.1 - - [24/Nov/2021 23:22:19] "GET /hello/Jim HTTP/1.1" 401 -
[2021-11-24 23:22:19,510] ERROR in app: Exception on /login [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/server.py", line 29, in login
    token = jwt.encode(payload, SECRET, algorithm="HS256").decode("utf8")
AttributeError: 'str' object has no attribute 'decode'
172.19.0.1 - - [24/Nov/2021 23:22:19] "POST /login HTTP/1.1" 500 -
172.19.0.1 - - [24/Nov/2021 23:22:19] "GET /ping HTTP/1.1" 401 -
[2021-11-24 23:22:19,576] ERROR in app: Exception on /login [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/server.py", line 29, in login
    token = jwt.encode(payload, SECRET, algorithm="HS256").decode("utf8")
AttributeError: 'str' object has no attribute 'decode'
172.19.0.1 - - [24/Nov/2021 23:22:19] "POST /login HTTP/1.1" 500 -

hamx0r avatar Nov 24 '21 23:11 hamx0r

Also, when running server.py locally (Python 3.6), I can POST to /login with the same curl command and get a token. It seems the test server code is fine, but it's not running as expected in the container (python 3.9).

hamx0r avatar Nov 24 '21 23:11 hamx0r

Changing Dockerfile's python version to 3.6 had no impact - it still throws error above on this line of code in server.py. Running the same tests using the same server.py running locally (not in container) work great. What's odd, though, is that each of the "Authenticated" tests in "hello" and "ping" result in an authentication POST request. I expected a single authentication request to get a token, and then that token to be used for all tests. I assume this is happening because Hello and Ping are in separate YAML files, so each one results in auth_stage web requests when imported.

It turns out, when installing locally, and just installing tavern, it results in PyJWT==1.7.1. However, the Dockerfile also adds in installing pyjwt which results in version 2.0.2 at the time of writing, and this version is resulting in the above errors. Here's a PR to fix this: https://github.com/taverntesting/tavern/pull/736

hamx0r avatar Nov 25 '21 04:11 hamx0r

great, @hamx0r ! You have found the key to the problem!

After testing, specifying the version of the third-party library in Dockerfile can avoid this problem.

dongfangtianyu avatar Nov 25 '21 05:11 dongfangtianyu

On other topics:

The Tavern tests are not run "in order"

Clearly, The Tavern tests are run "in order"

In test_hello.tavern.yaml, there are three steps:

  1. Unauthenticated /hello (in test_hello.tavern.yaml)
  2. Login and acquire token (in auth_parse.yaml)
  3. Authenticated /hello (in test_hello.tavern.yaml)

auth_parse.yaml is included and executed at the specified location https://github.com/taverntesting/tavern/blob/master/example/components/test_hello.tavern.yaml#L16

I expected a single authentication request to get a token, and then that token to be used for all tests.

Currently, the test files are isolated from each other, so the result of a request will not be shared among all tests. If you really need to do this, you can consider using a hook to directly share global variables in all requests, but it may make the test more complicated:

# conftest.py

test_login_token = ""


def pytest_tavern_beta_after_every_response(expected, response):
    global test_login_token
    if "token" in response.json():
        test_login_token = response.json()["token"]


def pytest_tavern_beta_before_every_test_run(test_dict, variables):
    variables["test_login_token"] = test_login_token

dongfangtianyu avatar Nov 25 '21 06:11 dongfangtianyu

Closing as the original issue seems to be fixed.

michaelboulton avatar Nov 18 '23 16:11 michaelboulton