Aiohttp slows down when using proxies

[makovez]

🐞 Describe the bug

When using proxies i notice that the lasts requests being made are very very very slow in comparision to the first ones. if i make 1000 requests it will start to slow down from around 900-1000 if i make 5000 requests it will start to slow down from around 4800-5000

💡 To Reproduce

Using any remote proxy (not local) I can provide the code and the proxy used service, if you dont have a proxy to try with, write me on telegram @soermejo

💡 Expected behavior

I expect it to be fast tiill the end without slowing down.

📋 Logs/tracebacks

import asyncio
import aiohttp

try:
    from asyncio.exceptions import TimeoutError
except ModuleNotFoundError:
    from concurrent.futures._base import TimeoutError

class Proxy:
    
    def __init__(self, timeout=15) -> None:
        self.timeout = timeout
        self.COUNT = 0
        timeout = aiohttp.ClientTimeout(sock_connect=timeout)
        self.session = aiohttp.ClientSession(connector=aiohttp.TCPConnector(ssl=False, limit=0), timeout=timeout)

    async def get(self, url, headers=None):
        http_proxy = "PROXYPOOL"
        try:
           
            async with self.session.get(url, headers=headers, proxy=http_proxy) as response:
                res = await response.text()
                self.COUNT +=1
                print(self.COUNT)
                return res

        except aiohttp.ClientError as e:
            print("Client error: ", e)
            return await self.get(url, headers)
            

        except TimeoutError as e:
            print("TimeoutError: ", e)
            return await self.get(url, headers)
            

async def main():
    url = "https://google.com"
    proxy = Proxy()

    tasks = []

    for _ in range(1000):
        tasks.append(asyncio.ensure_future(proxy.get(url)))
    
    await asyncio.gather(*tasks, return_exceptions=True)
    await proxy.session.close()



loop = asyncio.get_event_loop()

loop.run_until_complete(main())

📋 Your version of the Python

$ python --version
Python 3.8.5
...

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.7.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: [email protected]
License: Apache 2
Location: /home/namoso/.local/lib/python3.8/site-packages
Requires: typing-extensions, multidict, chardet, yarl, async-timeout, attrs
Required-by: proxybroker
...

$ python -m pip show multidict
Name: multidict
Version: 5.0.0
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/namoso/.local/lib/python3.8/site-packages
Requires: 
Required-by: yarl, aiohttp, sanic
...

$ python -m pip show yarl
Name: yarl
Version: 1.6.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/namoso/.local/lib/python3.8/site-packages
Requires: idna, multidict
Required-by: aiohttp
...

📋 Additional context

Im using linux

 IMPORTANT: aiohttp is both server framework and client library.
 For getting rid of confusing please put 'server', 'client' or 'both'
 word here.
 --> client

[makovez]

UPDATE: I've tried the same with httpx.

Aiohttp

• 5000 requests
• no concurrent limits
• same proxies source
• duration 1.38 min

Httpx

• 5000 requests
• no concurrent limits
• same proxies source
• duration 45 sec

I noticed that aiohttp is faster and less buggy at the startup (in fact im getting less errors) but at the end, the last requests made are very very slow. So this is deafinitely an aiohttp BUG.

[asvetlov]

Thanks for the report.

Perhaps it is because aiohttp closes the connection to a proxy after finishing a single request (keep-alive is not supported). https://github.com/aio-libs/aiohttp/blob/master/aiohttp/connector.py#L1047-L1050

This was added in 2014 after at the main job we inspected a lot of proxies that did not support keep-alive properly. Maybe it is not an issue now, I did not use proxies for 2.5 years and did not test the keep-alive problem since 2015.

[WisdomPill]

Hello!

Perhaps it is because aiohttp closes the connection to a proxy after finishing a single request (keep-alive is not supported). https://github.com/aio-libs/aiohttp/blob/master/aiohttp/connector.py#L1047-L1050

Can this be easily made a flag so that the connection can be reused?

[legout]

Hi, I found the same behavior and wonder, if there is a workaround or solution available.

[baby5]

Hi, I found the same behavior and wonder, if there is a workaround or solution available.

using httpx instead of