Change `requests` to `httpx`

[camilamaia]

Change requests to httpx https://github.com/encode/httpx

Features

HTTPX builds on the well-established usability of requests, and gives you:

• A broadly requests-compatible API.
• Standard synchronous interface, but with async support if you need it.
• HTTP/1.1 and HTTP/2 support.
• Ability to make requests directly to WSGI applications or ASGI applications.
• Strict timeouts everywhere.
• Fully type annotated.
• 99% test coverage.

Plus all the standard features of requests...

• International Domains and URLs
• Keep-Alive & Connection Pooling
• Sessions with Cookie Persistence
• Browser-style SSL Verification
• Basic/Digest Authentication
• Elegant Key/Value Cookies
• Automatic Decompression
• Automatic Content Decoding
• Unicode Response Bodies
• Multipart File Uploads
• HTTP(S) Proxy Support
• Connection Timeouts
• Streaming Downloads
• .netrc Support
• Chunked Requests

[tpiekarski]

Migrated recently HTTP requests from requests to aiohttp and looking forward to migrate some more requests to httpx this time. Would like to take a look and migrate these in the next week to prepare myself for coming sprint at EuroPython :) Did you start to change to httpx?

[camilamaia]

Migrated recently HTTP requests from requests to aiohttp and looking forward to migrate some more requests to httpx this time. Would like to take a look and migrate these in the next week to prepare myself for coming sprint at EuroPython :) Did you start to change to httpx?

This would be awesome! I didn't have the chance to start tackling this issue, so you can go ahead 🚀

If you have any troubles running it locally, please let me know. We are moving to poetry and soon it will be way easier to configure your development env.

[tpiekarski]

Migrated recently HTTP requests from requests to aiohttp and looking forward to migrate some more requests to httpx this time. Would like to take a look and migrate these in the next week to prepare myself for coming sprint at EuroPython :) Did you start to change to httpx?

This would be awesome! I didn't have the chance to start tackling this issue, so you can go ahead

That's great :)

If you have any troubles running it locally, please let me know. We are moving to poetry and soon it will be way easier to configure your development env.

Thanks, I keep that in mind.

[Pradhvan]

@tpiekarski are you working on this ? If not, I would be happy to take this up. 😄

[tpiekarski]

Please go ahead, I am out of order for some time. Thanks 😊

[Pradhvan]

@camilamaia on a quick overview of the issue I found requests to be used only in two files.

• scanapi/scanapi/tree/request_node.py : here I can try to find an quivalent of requests.requests for httpx

        response = requests.request(
            method,
            url,
            headers=self.headers,
            params=self.params,
            json=self.body,
            allow_redirects=False,
        )

• scanapi/scanapi/tree/testing_node.py:

import requests

requests is being imported but it isn't being used. But the linter is not showing it as an unused import? Am I missing something here?

Besides those, a couple of test files and of what I saw for tests requests is only used for GET. So for tests, it would be an easy replacement I am assuming.

• scanapi/tests/unit/test_hide_utils.py
• scanapi/tests/unit/test_scan.py
• scanapi/tests/unit/test_utils.py
• scanapi/tests/unit/evaluators/test_code_evaluator.py

[marcuxyz]

@camilamaia on a quick overview of the issue I found requests to be used only in two files.

• scanapi/scanapi/tree/request_node.py : here I can try to find an quivalent of requests.requests for httpx

        response = requests.request(
            method,
            url,
            headers=self.headers,
            params=self.params,
            json=self.body,
            allow_redirects=False,
        )

• scanapi/scanapi/tree/testing_node.py:

import requests

requests is being imported but it isn't being used. But the linter is not showing it as an unused import? Am I missing something here?

Besides those, a couple of test files and of what I saw for tests requests is only used for GET. So for tests, it would be an easy replacement I am assuming.

• scanapi/tests/unit/test_hide_utils.py
• scanapi/tests/unit/test_scan.py
• scanapi/tests/unit/test_utils.py
• scanapi/tests/unit/evaluators/test_code_evaluator.py

to remove imports and unused variables. I recommend:

https://pypi.org/project/autoflake/#:~:text=To%20remove%20all%20unused%20imports,remove-unused-variables%20option.

[camilamaia]

I tested changing requests to httpx, but we have one problem. Curlify does not work with httpx:

Loading file ../examples/demo-api/.scanapi.yaml
Loading file ../examples/demo-api/scanapi.yaml
Writing documentation
Making request GET http://demo.scanapi.dev/api/health/
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/__main__.py", line 62, in run
    scan()
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/scan.py", line 50, in scan
    write_report(results)
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/scan.py", line 63, in write_report
    reporter.write(results)
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/reporter.py", line 27, in write
    content = render(template_path, context, has_external_template)
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/template_render.py", line 13, in render
    return chosen_template.render(**context)
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/Users/camilamaia/workspace/scanapi-org/scanapi/scanapi/templates/html.jinja", line 408, in top-level template code
    <p>{{request|curlify}}</p>
  File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/curlify.py", line 25, in to_curl
    if request.body:
AttributeError: 'Request' object has no attribute 'body'

https://github.com/ofw/curlify/blob/master/curlify.py#L27

I only changed this to this:

import httpx
....
response = httpx.request(
            method,
            url,
            headers=self.headers,
            params=self.params,
            json=self.body,
            allow_redirects=False,
        )

If I remove this line, everything works properly.

About the differences between requests and httpx: https://www.python-httpx.org/compatibility/

[marcorichetta]

File "/Users/camilamaia/Library/Caches/pypoetry/virtualenvs/scanapi-KvDFA3Cq-py3.7/lib/python3.7/site-packages/curlify.py", line 25, in to_curl if request.body: AttributeError: 'Request' object has no attribute 'body'

https://github.com/ofw/curlify/blob/master/curlify.py#L27

I only changed [this](https://github.com/scanapi/scanapi/blob/97c4451c6b4a52d13f32c617053bbfe8cecdc75f/scanapi/tree/request_node.py#L99) to this:

```python
import httpx
....
response = httpx.request(
            method,
            url,
            headers=self.headers,
            params=self.params,
            json=self.body,
            allow_redirects=False,
        )

If I remove this line, everything works properly.

It seems curlify doesn't support httpx at the moment, specifically:

httpx does not have a .body but contents can be accessed via .read() instead.

I think the best option is to use this enhanced (and updated) version of curlify https://github.com/marcuxyz/curlify2, otherwise it'll be needed to maybe fork (?) curlify or modify it's use inside this project. https://github.com/scanapi/scanapi/blob/eba2ecd0f1067fee364e472b606a6c54cee27271/scanapi/reporter.py#L23

[camilamaia]

@marcorichetta great catch. And in fact we are already using curlify2:

https://github.com/scanapi/scanapi/blob/master/pyproject.toml#L23

Actually, @marcuxyz created this curlify2 exactly for this reason!

[marcorichetta]

@marcorichetta great catch. And in fact we are already using curlify2:

https://github.com/scanapi/scanapi/blob/master/pyproject.toml#L23

Actually, @marcuxyz created this curlify2 exactly for this reason!

Oh, I read https://github.com/scanapi/scanapi/issues/208#issuecomment-673602067 and thought it was unresolved.

So this issue is already solved by PR https://github.com/scanapi/scanapi/pull/330, right?

[camilamaia]

@marcorichetta yes, exactly, the curl issue was solved. But we are not using httpx yet. We are still using requests.

[taconi]

@Pradhvan are you working on this ? If not, I would be happy to take this up

[Pradhvan]

@Pradhvan are you working on this ? If not, I would be happy to take this up

no, feel free to take it. 😄