Duplicate requests in latest_requests() if there are chunks

[VelorumS]

Was fine with 1.0.5, does duplicates in 1.1.0.

The change: https://github.com/gabrielfalcao/HTTPretty/commit/b6161cebfdaf5f68ef49526e51b460b8e0c0c6c2#r50812634

-            cls.latest_requests[-1] = request
+            pos = cls.latest_requests.index(request)
+            cls.latest_requests[pos] = request

httpretty.historify_request() is called multiple times for the same request.

From here:

  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.6/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1081, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.6/http/client.py", line 1002, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.6/dist-packages/httpretty/core.py", line 683, in sendall

And here:

  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.6/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 1002, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.6/dist-packages/httpretty/core.py", line 699, in sendall

[ThiefMaster]

These duplicate requests broke my CI. Pinning httpretty<1.1 is a workaround but a fix would be appreciated.

[Gidgidonihah]

I also came looking for this because of a broken CI.

[gabrielfalcao]

Oh gee, thanks @ChipmunkV for the solution. I'll get a new release out soon and will ping you folks once it's done.

[gabrielfalcao]

@ChipmunkV, @Gidgidonihah , @ThiefMaster I'm attempting to write a functional test to reproduce the bug. Would it be possible for you to share some example code to reproduce the issue?

[Gidgidonihah]

FYI, I cloned master after merge and tested it, and my tests are still failing with the wrong number of requests in latest_requests. I don't yet have a MWE example, but I would love to get one worked up.

[gabrielfalcao]

@Gidgidonihah I'd love to work with you in a solution for this, I just want to make sure there is an automated test to reproduce the issue.

I tried running the tests from indico-plugins-cern that @ThiefMaster mentioned but ran into some issues and ended up spending too much time, so I gave up on running the test. I also had trouble understanding what these tests are really trying to accomplish: https://github.com/indico/indico-plugins-cern/blob/03acd644ab2476b37c693250f4ea62b9f26985c1/ravem/tests/operations/connect_test.py#L62-L70

My main concern with httpretty.latest_requests() is whether it have been storing requests incorrectly up to the release 1.0.5. I that is the case, then I'm afraid that you, @ThiefMaster and anyone else who relies on len(httpretty.latest_requests()) might have been relying on an HTTPretty bug rather than a feature.

I take automated testing very seriously and want to make sure that whatever the case is, we can find the best possible solution for HTTPretty and ensure that your tests pass correctly (without false-positives).

How do you think I should proceed with this issue?

[gabrielfalcao]

The issue title suggests that the problem occurs "if there are chunks". I suppose this is a problem with chunked requests, is that the case for everyone here? Would any of you help me reproducing the issue?

[ThiefMaster]

Feel free to send a PR against my repo that reverts this commit and installs httpretty from git.

That way you don't need to setup the somewhat complex environment locally but still see if the error happens or not.

Alternatively, if you point me to the branch from which I can install the (potentially) fixed version I can give it a try locally.

[gabrielfalcao]

Thanks @ThiefMaster, I already merged 0eb4b169ad8d5ff7bda4afaef7a9249cf70e758d and released the changes in version 1.1.3 if you could kindly test it :)

[gabrielfalcao]

@Gidgidonihah if you could also test your code against 1.1.3 that would be great. Thanks folks!

[Gidgidonihah]

@gabrielfalcao Same result for me. Thanks for getting on this quickly. I'll try to get an MWE together at the end of my sprint.

[Gidgidonihah]

@gabrielfalcao Here's a gist for a MWE.

Using a python 3.9.5 virtualenv, just ran:

pip install -r requirements.txt
python -m unittest

[tim-schilling]

@gabrielfalcao It looks like there's another thing duplicating requests (I think). I've created a unit test within the repo that fails.

I narrowed it down to a POST that has data. Unless it's a preflight request that's being stored? That would explain the body being 0, though I'd expect the method to be OPTIONS and not POST.

[daveygit2050]

@gabrielfalcao I can confirm that it only appears to impact POST requests as @tim-schilling suggested:

import httpretty
import requests

httpretty.enable()

httpretty.register_uri(method=httpretty.POST, uri="https://example.com/foo")
requests.post("https://example.com/foo", data="bar")
len(httpretty.latest_requests())
# returns 2

httpretty.reset()

httpretty.register_uri(method=httpretty.GET, uri="https://example.com/foo")
requests.get("https://example.com/foo")
len(httpretty.latest_requests())
# returns 1

[gabrielfalcao]

@gabrielfalcao I can confirm that it only appears to impact POST requests as @tim-schilling suggested:

import httpretty
import requests

httpretty.enable()

httpretty.register_uri(method=httpretty.POST, uri="https://example.com/foo")
requests.post("https://example.com/foo", data="bar")
len(httpretty.latest_requests())
# returns 2

httpretty.reset()

httpretty.register_uri(method=httpretty.GET, uri="https://example.com/foo")
requests.get("https://example.com/foo")
len(httpretty.latest_requests())
# returns 1

Thanks @daveygit2050 I'll incorporate this snippet into a new test case to reproduce the issue as soon as I can manage. Unfortunately I might not be able to attend to this until next week but I'll keep yall posted.

[ThiefMaster]

A release containing this fix would be great!

[enthooz]

Thank you for creating a tool that helps improve automated tests! 🤗

I recently encountered this issue and wanted to add more info. In my testing, this applies to any requests (POST, PUT, DELETE, and/or PATCH) when they contain a request body. Below is a test file that demonstrates this behavior. (I did not include tests with request bodies for HTTP methods which do not allow request bodies (GET, HEAD, CONNECT, and OPTIONS), however these also demonstrate the behavior when a request body is provided.)

import httpretty
import urllib.request as request


class TestHTTpretty:
    @httpretty.activate(verbose=True, allow_net_connect=False)
    def run(self, method, data=None):
        url = 'https://example.com/foo'
        httpretty.register_uri(method=method, uri=url)
        req = request.Request(url=url, method=method, data=data)
        request.urlopen(req)
        assert len(httpretty.latest_requests()) == 1

    # the first eight tests with empty request bodies succeed
    def test_get(self):
        self.run(httpretty.GET)

    def test_head(self):
        self.run(httpretty.HEAD)

    def test_connect(self):
        self.run(httpretty.CONNECT)

    def test_options(self):
        self.run(httpretty.OPTIONS)

    def test_post_empty(self):
        self.run(httpretty.POST)

    def test_put_empty(self):
        self.run(httpretty.PUT)

    def test_delete_empty(self):
        self.run(httpretty.DELETE)

    def test_patch_empty(self):
        self.run(httpretty.PATCH)

    # the following four tests with populated request bodies fail
    def test_post_with_data(self):
        self.run(httpretty.POST, b'lorem ipsum')

    def test_put_with_data(self):
        self.run(httpretty.PUT, b'lorem ipsum')

    def test_delete_with_data(self):
        self.run(httpretty.DELETE, b'lorem ipsum')

    def test_patch_with_data(self):
        self.run(httpretty.PATCH, b'lorem ipsum')

Output:

============================ short test summary info ============================
FAILED test/httpretty_test.py::TestHTTpretty::test_post_with_data - assert 2 == 1
FAILED test/httpretty_test.py::TestHTTpretty::test_put_with_data - assert 2 == 1
FAILED test/httpretty_test.py::TestHTTpretty::test_delete_with_data - assert 2 == 1
FAILED test/httpretty_test.py::TestHTTpretty::test_patch_with_data - assert 2 == 1

[cansin]

I am dealing with the exact same issue. Looks like there was a fix merged into the master. As @ThiefMaster was pointing out, @gabrielfalcao would it be possible to deploy a new release?

[chassing]

+1 for a 1.1.5 release

[ThiefMaster]

@gabrielfalcao Can we please get an 1.1.5 release containing this fix?

[gabrielfalcao]

Salutations all,

my macbook got damaged and I'm quite a bit limited at present due to the time taken for the new parts to arrive. I plan on applying the patch and resolve the situation as soon as the universe collaborates. Please hang in there and I'll report back as soon as humanely possible.

Kind regards, Gabriel

[gabrielfalcao]

Hi folks, I presently find myself somewhat able to work on this issue. I'll do what I can to make a release. Thanks for your patience so far.

[chassing]

Hi @gabrielfalcao,

This is a friendly reminder; I hope you'll find some time to make a new release :)

Thank you

[sfuhrm]

Problem exists in httpretty==1.1.4 httpretty==1.0.5 works as a workaround

@gabrielfalcao thanks for your great effort, it is appreciated!