aiohttp - concurrent requests are getting hung

[raees-khan]

Concurrent requests are getting hung. Here's a sample code that I am using to test concurrent requests.

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        print(await response.text())

async def main(n):
    url = "https://httpstat.us/200"
    async with aiohttp.ClientSession() as session:
        tasks = [asyncio.create_task(fetch(session, url)) for _ in range n]
        await asyncio.gather(*tasks)

asyncio.run(main(10))

When I make 10 concurrent requests, the first 4-5 requests are made concurrently and then it gets hung for over 10 seconds and then starts the remaining tasks which get hung again after running 2-3 concurrent requests. If I make 100 concurrent requests, it makes about 25-30 concurrent requests and gets hung and then makes 5-6 requests and gets hung again, it does this until all tasks are complete.

It's taking over two minutes to make 100 requests to https://httpstat.us/200 with aiohttp.

If I don't use a persistent ClientSession and create new ClientSession for every request, then all hundred requests finish within 5 seconds without getting hung.

I am not sure what I am doing here. Any help will be highly appreciated.

I am running Python 3.7.2 and aiohttp 3.5.4

[socketpair]

First, Task is not required. Second, I have tried and everything works as a charm without any hangs.

import aiohttp
import asyncio

async def fetch(session):
    async with session.get("https://httpstat.us/200") as response:
        print(await response.text())

async def main(n):
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*(fetch(session) for _ in range(n)))

asyncio.run(main(100))

I've checked exactly your example and everything is the same: no hangs. Please check your firewall/gateway

[socketpair]

Feel free to reopen issue. I'm closing now since can not be reproduced.

[raees-khan]

Feel free to reopen issue. I'm closing now since can not be reproduced.

@socketpair I've ruled out the possibility of firewall/gateway issues. Moreover, I don't see any hangs if I create a new ClientSession for every request.

With this code, I never see concurrent requests. That is, after 20-30 requests, the program just hangs.

import aiohttp
import asyncio

async def fetch(session):
    async with session.get("https://httpstat.us/200") as response:
        print(await response.text())

async def main(n):
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*(fetch(session) for _ in range(n)))

asyncio.run(main(100))

Following works like charm. No hangs!!

import aiohttp
import asyncio

async def fetch(session):
    async with aiohttp.ClientSession() as session:
        async with session.get("https://httpstat.us/200") as response:
        print(await response.text())

async def main(n):
    await asyncio.gather(*(fetch() for _ in range(n)))

asyncio.run(main(100))

I am also seeing a frequent SSL errors which I believe is a known bug with aiohttp 3.5.4 on Python 3.7.x. However, I don't think the code hangs because of the SSL errors.

SSL error in data received
handle_traceback: Handle created at (most recent call last):
  File "concurrent_test.py", line 406, in <module>
    asyncio.run(main(), debug=True)
  File "/anaconda3/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/anaconda3/lib/python3.7/asyncio/base_events.py", line 560, in run_until_complete
    self.run_forever()
  File "/anaconda3/lib/python3.7/asyncio/base_events.py", line 528, in run_forever
    self._run_once()
  File "/anaconda3/lib/python3.7/asyncio/base_events.py", line 1756, in _run_once
    handle._run()
  File "/anaconda3/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/anaconda3/lib/python3.7/asyncio/selector_events.py", line 716, in _add_reader
    self._loop._add_reader(fd, callback, *args)
  File "/anaconda3/lib/python3.7/asyncio/selector_events.py", line 260, in _add_reader
    handle = events.Handle(callback, args, self, None)
protocol: <asyncio.sslproto.SSLProtocol object at 0x1080d1e10>
transport: <_SelectorSocketTransport fd=12 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/anaconda3/lib/python3.7/asyncio/sslproto.py", line 526, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/anaconda3/lib/python3.7/asyncio/sslproto.py", line 207, in feed_ssldata
    self._sslobj.unwrap()
  File "/anaconda3/lib/python3.7/ssl.py", line 767, in unwrap
    return self._sslobj.shutdown()
ssl.SSLError: [SSL: KRB5_S_INIT] application data after close notify (_ssl.c:2592)

When I use a persistent session to make concurrent requests, it sometimes hangs and completes after a very long time and sometimes it fails with following traceback. I am not sure why asyncio.TimeoutError is being raised.

Traceback (most recent call last):
  File "concurrent_test.py", line 406, in <module>
    asyncio.run(main(), debug=True)        
  File "/anaconda3/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/anaconda3/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "concurrent_test.py", line 386, in main
    await asyncio.gather(*tasks)
  File "concurrent_test.py", line 248, in fetch
    async with session.post(url, **post_data) as response:
  File "/anaconda3/lib/python3.7/site-packages/aiohttp/client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "/anaconda3/lib/python3.7/site-packages/aiohttp/client.py", line 575, in _request
    break
  File "/anaconda3/lib/python3.7/site-packages/aiohttp/helpers.py", line 585, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError

[renanvieira]

@socketpair looks like this SSL error is a bug that affects aiohttp/asyncio on python3.7. No fix yet.

More info: #3535

[yunstanford]

Observing same hang issue

[MDziwny]

I have the same problem, even when opening a new ClientSession:

ssl.SSLError: [SSL: KRB5_S_INIT] application data after close notify (_ssl.c:2592)

If needed, I can provide more information.

[ashkan-leo]

Observing the same issue

[sadikekin]

Having the same issue.

[sertraline]

Same issue.

Using one persistent session (ssl=False) for entire script: aiohttp makes 10 concurrent connections, then hangs, then makes random quantity of concurrent requests. Notice the timing:

Image

26 requests complete in about 5 seconds. Now I do what @raees-khan suggested, creating new session for every request:

Image

Script finishes in less than a second (4,60s vs 0.60s).

[dassencio]

The main problem with creating a single ClientSession for each GET request (as suggested by @raees-khan) is that you lose the ability to control the maximum number of simultaneous connections created by aiohttp.

In my case (Python 3.8.2), a single GET request to https://diego.assencio.com already caused aiohttp to get stuck (HTTPS requests to other URLs worked fine, though). By simply creating a ClientSession object using the default parameters, the requests started succeeding, but a lot slower than usual. To clarify, I changed this:

    async with ClientSession(connector=TCPConnector(limit=16)) as session:
        ...

into this:

    async with ClientSession() as session:
        ...    

and the result was "requests succeeding but slowly", compared to "not a single request succeeding".

Interestingly, the idea from @raees-khan produced the exact same outcome for me: requests started succeeding, but just as slowly as with a single instance of ClientSession created using the default parameters.

For the record, HTTP requests are working fine. All issues I observed happen with HTTPS only.

[samip5]

I'm also hitting this with my matrix bot, it will work for a long time and then crash with:

Traceback (most recent call last):
  File "main.py", line 98, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "main.py", line 86, in main
    await client.sync_forever(full_state=True)
  File "/home/sm/Github/dankbot/venv/lib/python3.7/site-packages/nio/client/async_client.py", line 1147, in sync_forever
    await self.run_response_callbacks([await response])
  File "/usr/lib/python3.7/asyncio/tasks.py", line 533, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/sm/Github/dankbot/venv/lib/python3.7/site-packages/nio/client/async_client.py", line 1003, in sync
    timeout,
  File "/home/sm/Github/dankbot/venv/lib/python3.7/site-packages/nio/client/async_client.py", line 705, in _send
    method, path, data, headers, trace_context, timeout,
  File "/home/sm/Github/dankbot/venv/lib/python3.7/site-packages/nio/client/async_client.py", line 252, in wrapper
    return await func(self, *args, **kwargs)
  File "/home/sm/Github/dankbot/venv/lib/python3.7/site-packages/nio/client/async_client.py", line 781, in send
    else timeout,
  File "/home/sm/Github/dankbot/venv/lib/python3.7/site-packages/aiohttp/client.py", line 544, in _request
    await resp.start(conn)
  File "/home/sm/Github/dankbot/venv/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 905, in start
    self._continue = None
  File "/home/sm/Github/dankbot/venv/lib/python3.7/site-packages/aiohttp/helpers.py", line 656, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError

Seperate clientSession per request:

    async def _lab(self):
        try:
            topic = self.args[0]
            if topic == "light":
                msg = await self._lab_light()
                await send_text_to_room(self.client, self.room.room_id, msg)
            elif topic == "wifi":
                msg = await self._lab_wifi()
                await send_text_to_room(self.client, self.room.room_id, msg)
            else:
                first_msg = "You did not specify which information you want, so I'm gonna guess it as both."
                await send_text_to_room(self.client, self.room.room_id, first_msg)
                light_msg = await self._lab_light()
                await send_text_to_room(self.client, self.room.room_id, light_msg)
                wifi_msg = await self._lab_wifi()
                await send_text_to_room(self.client, self.room.room_id, wifi_msg)
        except IndexError:
            pass


    async def _lab_light(self):
        async with aiohttp.ClientSession() as session:
            async with session.get("http://localhost/pi_api/gpio/?a=readPin&pin=1") as response:
                api_response = await response.json()
                api_data = api_response['data']
                if api_data == 0:
                    msg = "Someone is probably at the lab as the light is on."
                else:
                    msg = "Nobody is at the lab, the light is currently off."
                return msg

    async def _lab_wifi(self):
        async with aiohttp.ClientSession() as session:
            async with session.get("http://localhost/visitors/api/v1/visitors.php?format=json") as response:
                api_response = await response.json()
                if not api_response:
                    msg = "Nobody is using the Wifi that I know of currently."
                else:
                    msg = "Someone or multiple someones are using the Wifi at the lab currently."
                return msg

[Dreamsorcerer]

I can't see any relation between the code you've posted and that traceback.

The traceback suggests that the timeout occurred within the nio library. I'd assume if you are dealing with a federated service, then timeouts could be pretty common, so not sure how that library is supposed to deal with this. It appears there is a config option to change the timeout parameter though: https://github.com/poljar/matrix-nio/blob/master/nio/client/async_client.py#L300

[kevinl1210]

I have the same issue. I am making 10 concurrent requests with asyncio.gather(), the first request is successful but the rest are hanging and timed out. I running the script on WSL ubuntu 18.04.2, python 3.8.0, aiohttp 3.7.4.

However, if I run the same script on the local windows 10 machine, it is working fine.

[evgenybf]

I've got the same issue on Ubuntu 18.04. It's not safe to use ClientSession from several coroutines asyncroneously (gather()-ing them or from Tasks). For examle, concurent coroutines that scrape different sites may start receiving other "co-worker" responses. It happens more often when majoritiy of the sites do redirect and close connection or just respond with a delay. It can be resolved by creating a separate session instance for each worker. I guess, that libraries that utilize aiohttp client (e.g. gcloud-aio-pubsub - SubscriberClient class) may saffer from the same issue.

[parafoxia]

So I've been having these issues for ages, and it seems, at least in my exact use-case, to only affect WSL. I've tried to trace the cause of the issue as far as I can, but I can't seem to work out if it goes any further. This is for 3.8.1 -- I would try with 4.0.0a but it physically will not install no matter what I do. This is from a session.request call:

https://github.com/aio-libs/aiohttp/blob/v3.8.1/aiohttp/client.py#L353 https://github.com/aio-libs/aiohttp/blob/v3.8.1/aiohttp/client.py#L535 https://github.com/aio-libs/aiohttp/blob/v3.8.1/aiohttp/connector.py#L542 https://github.com/aio-libs/aiohttp/blob/v3.8.1/aiohttp/connector.py#L907 https://github.com/aio-libs/aiohttp/blob/v3.8.1/aiohttp/connector.py#L1175 https://github.com/aio-libs/aiohttp/blob/v3.8.1/aiohttp/connector.py#L986

That's as far as I can trace before VSCode and GitHub can only find stuff in pyi or test files respectively. If anyone can do a better job at tracing that, let me know.

I'm kinda hoping my time doing this is wasted and it'll be fixed in v4.0.0a anyways, but just leaving this here for further investigation.

[ughstudios]

This is also happening for me in 2022

[datdinhquoc]

Me too, the same on Ubuntu WSL (desktop) & Centos 7 (aws) when using aiohttp to make web server, even print (first line in request handler) hangs for a while after 5 times browser make requests to a handler which has a call to another server and that call is timed-out. Other handlers which have no network connections (no aiohttp.ClientSession to another server) are still fine.

Tried another client httpx but it's the same after 6 timeouts, something not bcoz of aiohttp.

[datdinhquoc]

Found out my case, it's browser's 6 pending reqs / server limit, not bcoz of aiohttp