Issue using big response object in Entry with async client

[Veritogen]

Hey, firstly: big thanks for the library and the effort you put into it. It really helps us while testing.

While testing some things in our code base, I discovered that mocked async calls never finish if the response object passed to the Entry is above some size.

I wrote some code to be able to reproduce the behaviour:

import asyncio
import datetime
import json

import httpx
from mocket import mocketize
from mocket.mockhttp import Entry


@mocketize(strict_mode=True)
def test_sync_case():

    test_uri = "https://abc.de/testdata/"
    base_timestamp = int(datetime.datetime.now().timestamp())
    response = [
        {"timestamp": base_timestamp + i, "value": 1337 + 42 * i} for i in range(30_000)
    ]
    Entry.single_register(
        method=Entry.POST,
        uri=test_uri,
        body=json.dumps(
            response,
        ),
        headers={"content-type": "application/json"},
    )

    with httpx.Client() as client:
        response = client.post(test_uri)
        print("high number sync: ", len(response.json()))


@mocketize(strict_mode=True)
def test_async_case_low_number():

    test_uri = "https://abc.de/testdata/"
    base_timestamp = int(datetime.datetime.now().timestamp())
    response = [
        {"timestamp": base_timestamp + i, "value": 1337 + 42 * i} for i in range(100)
    ]
    Entry.single_register(
        method=Entry.POST,
        uri=test_uri,
        body=json.dumps(
            response,
        ),
        headers={"content-type": "application/json"},
    )

    async def main():
        async with httpx.AsyncClient() as client:
            response = await client.post(test_uri)

        print("low number async: ", len(response.json()))

    loop = asyncio.new_event_loop()
    loop.run_until_complete(main())


@mocketize(strict_mode=True)
def test_async_case_high_number():

    test_uri = "https://abc.de/testdata/"
    base_timestamp = int(datetime.datetime.now().timestamp())
    response = [
        {"timestamp": base_timestamp + i, "value": 1337 + 42 * i} for i in range(30_000)
    ]
    Entry.single_register(
        method=Entry.POST,
        uri=test_uri,
        body=json.dumps(
            response,
        ),
        headers={"content-type": "application/json"},
    )

    async def main():
        async with httpx.AsyncClient() as client:
            # Awaiting the response never finishes
            response = await client.post(test_uri)

        print("high number async: ", len(response.json()))

    loop = asyncio.new_event_loop()
    loop.run_until_complete(main())


test_sync_case()
test_async_case_low_number()
test_async_case_high_number()

While the functions test_sync_case() and test_async_case_low_number() run through, test_async_case_high_number() never finishes. The number until the code works for test_async_case_low_number() is response = [{"timestamp": base_timestamp + i, "value": 1337 + 42 * i} for i in range(1_525)]. At 1526 it stops working.

Am I doing something wrong? Or could a bug be hiding here?

We're running Python 3.11 with httpx==0.24.1 and mocket==3.12.3

[mindflayer]

Sorry, I made a mistake (hence deleted my answers). Let me have a deeper look at it.

[Veritogen]

@mindflayer thank you for the fast reply. Much appreciated. Let me know if you need any help troubleshooting or getting to the root of the issue :)

[mindflayer]

Hi @Veritogen, I spent some time on it but still had no luck. Feel free to investigate further, I'll try to have another look at it in the next few days, in case you don't find anything interesting earlier.

[mindflayer]

I found the problem and I am trying to fix a test that breaks as a drawback. I am positive, btw. :)

[mindflayer]

Still having issues with aiohttp, a few tests are now broken. I wonder if @ento could have a look at https://github.com/mindflayer/python-mocket/pull/226. For the moment forgetting about the "not closing all the sockets" issue.

[mindflayer]

a test that breaks

A few tests that break, actually. :/

[ento]

Left a comment with my findings in #226 - hope that's useful.

[mindflayer]

Hi @Veritogen, I know the issue was opened months ago, but I managed to make it work in my last PR. I still have issues with aiohttp, but only with Python versions older than 3.11. I am wondering if it's time give up with using it to test Mocket. It used to be the main client I used, but then it started giving me a lot of troubles I don't experience with httpx...

[Veritogen]

Thats great to hear. We still kept track and actually were about to put some of our resources into fixing this issue. If I recall correctly we switched from aiohttp to httpx as well due to troubles we experienced.

[mindflayer]

I've just released a new version which should work for your test-case: https://pypi.org/project/mocket/3.12.7/

[mindflayer]

I'm closing this issue, feel free to comment in case you still experience problems.

[Veritogen]

I can confirm that it works. Sorry for taking so long to respond and many, many thanks for fixing the issue. Much appreciated!