responses icon indicating copy to clipboard operation
responses copied to clipboard

responses multipart_matcher does not consume the passed iterator; how to mock?

Open kfot opened this issue 9 months ago • 2 comments

Describe the bug

Hi, I am developing some python code utilizing mulipart/form-data headers for uploading a file. When I try to mock the endpoint response with the following code, then it hangs.

@responses.activate
def test_multipart_upload(client, test_file_definition):
    responses.add(
        responses.POST,
        url=gen_url(<some_endpoint>),
        json=1,
    )
    client.multipart_upload(...)

Responses: 0.25.0 Python: 3.11

Additional context

Once I will add a multipart matcher, I can get an error stating that the iterator does not match the data (as the generator is just an object in the memory).

~#@❯ pytest -x --no-cov -k test_multipart_upload
Test session starts (platform: win32, Python 3.11.8, pytest 7.4.0, pytest-sugar 1.0.0)
rootdir: C:\Users\<REDACTED>
configfile: pytest.ini
testpaths: tests
plugins: anyio-4.3.0, nbmake-1.5.3, cov-4.1.0, icdiff-0.9, sugar-1.0.0, timeout-2.3.1


―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_create_new_version ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― 

client = Client(dummy@unknown(http://test:54422))>
tmp_path = WindowsPath('C:/Users/<REDACTED>/pytest-442/test_multipart_upload0')

    @responses.activate
    def test_multipart_upload(client, tmp_path):
        file_path = tmp_path / "test_data.txt"
        write_file(file_path)  # 1 kB

        responses.add(
            responses.POST,
            url=gen_url(<some_endpoint>),
            match=[
                responses.matchers.multipart_matcher({"file_name": b""})
                ],
            json=1,
        )
>       client.multipart_upload(...)

C:\Users\<REDACTED>\tests\client\test_multipart_upload.py
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
C:\Users\<REDACTED>\envs\mylib311\Lib\site-packages\responses\__init__.py:1173: in send
    return self._on_request(adapter, request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <responses.RequestsMock object at 0x000002AAF71E2B10>, adapter = <requests.adapters.HTTPAdapter object at 0x000002AAF743D210>, request = <PreparedRequest [POST]>, retries = None
kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': None, ...}, match = None
match_failed_reasons = ['multipart/form-data doesn\'t match. Request body differs. <generator object chunk_file_for_upload at 0x000002AAF73C0...: form-data; name="file_name"; filename="file_name"\\r\\n\\r\\n\\r\\n--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1--\\r\\n\'']
resp_callback = None
error_msg = 'Connection refused by Responses - the call doesn\'t match any registered mock.\n\nRequest: \n- POST http://test:54422... form-data; name="file_name"; filename="file_name"\\r\\n\\r\\n\\r\\n--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1--\\r\\n\'\n'
i = 0, m = <Response(url='http://test:54422/<some_endpoint>' status=200 content_type='application/json' headers='null')>

    def _on_request(
        self,
        adapter: "HTTPAdapter",
        request: "PreparedRequest",
        *,
        retries: Optional["_Retry"] = None,
        **kwargs: Any,
    ) -> "models.Response":
        # add attributes params and req_kwargs to 'request' object for further match comparison
        # original request object does not have these attributes
        request.params = self._parse_request_params(request.path_url)  # type: ignore[attr-defined]
        request.req_kwargs = kwargs  # type: ignore[attr-defined]
        request_url = str(request.url)

        match, match_failed_reasons = self._find_match(request)
        resp_callback = self.response_callback

        if match is None:
            if any(
                [
                    p.match(request_url)
                    if isinstance(p, Pattern)
                    else request_url.startswith(p)
                    for p in self.passthru_prefixes
                ]
            ):
                logger.info("request.allowed-passthru", extra={"url": request_url})
                return self._real_send(adapter, request, **kwargs)  # type: ignore

            error_msg = (
                "Connection refused by Responses - the call doesn't "
                "match any registered mock.\n\n"
                "Request: \n"
                f"- {request.method} {request_url}\n\n"
                "Available matches:\n"
            )
            for i, m in enumerate(self.registered()):
                error_msg += "- {} {} {}\n".format(
                    m.method, m.url, match_failed_reasons[i]
                )

            if self.passthru_prefixes:
                error_msg += "Passthru prefixes:\n"
                for p in self.passthru_prefixes:
                    error_msg += f"- {p}\n"

            response = ConnectionError(error_msg)
            response.request = request

            self._calls.add(request, response)
>           raise response
E           requests.exceptions.ConnectionError: Connection refused by Responses - the call doesn't match any registered mock.
E
E           Request: 
E           - POST http://test:54422/<some_endpoint>?complete=False&encoding=utf-16
E
E           Available matches:
E           - POST http://test:54422/<some_endpoint> multipart/form-data doesn't match. Request body differs. <generator object chunk_file_for_upload at 0x000002AAF73C0DC0> aren't equal b'--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1\r\nContent-Disposition: form-data; name="file_name"; filename="file_name"\r\n\r\n\r\n--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1--\r\n'

C:\Users\<REDACTED>\envs\mylib311\Lib\site-packages\responses\__init__.py:1100: ConnectionError

Version of responses

0.25.0

Steps to Reproduce

Add an generator to be consumed by the request.

Expected Result

Code using responses does not hang anymore.

Actual Result

Tests using responses hang.

kfot avatar May 13 '24 16:05 kfot