curl_cffi
curl_cffi copied to clipboard
[BUG] Unable to make simultaneous asynchronous requests
Describe the bug
Making multiple simultaneous requests to a single host results in a curl
error if a TCP connection to the requested host was established earlier (a request was made previously).
To Reproduce
- Install
httpx
usingpip install httpx
- Install
curl-cffi
usingpip install curl-cffi
- Run this code:
import asyncio
import sys
from curl_cffi.requests import AsyncSession
from httpx import AsyncClient
BASE_URL = 'https://images.pexels.com'
PATHS = ['/photos/842711/pexels-photo-842711.jpeg'] * 2
async def test_httpx():
print('----- httpx -----')
async with AsyncClient() as session:
await session.get(BASE_URL)
tasks = [session.get(BASE_URL + path) for path in PATHS]
print(await asyncio.gather(*tasks))
async def test_curl_cffi():
print('----- curl-cffi -----')
async with AsyncSession() as session:
await session.get(BASE_URL)
tasks = [session.get(BASE_URL + path) for path in PATHS]
print(await asyncio.gather(*tasks))
if __name__ == '__main__':
if sys.platform == 'win32':
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
asyncio.run(test_httpx())
asyncio.run(test_curl_cffi())
Expected behavior
The requests will be successfully completed and the bytes of the requested images will be received with an HTTP status of 200, as occurs when using httpx
.
Versions
- OS: Windows 11
- curl_cffi version: 0.6.2
-
pip freeze
dump:
anyio==4.3.0
certifi==2024.2.2
cffi==1.16.0
curl_cffi==0.6.2
exceptiongroup==1.2.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
idna==3.7
pycparser==2.22
sniffio==1.3.1
typing_extensions==4.11.0
Additional context
- Which session are you using? Async.
- If using async session, which loop implementation are you using? Asyncio.
A temporary fix is to force http/1.1, I'm still investigating a further solution.
- async with AsyncSession() as session:
+ async with AsyncSession(http_version=CurlHttpVersion.V1_1) as session:
Hello! Are there any new details on the issue?
Hello again, any updates?
same error. when streaming curl_cffi.requests.errors.RequestsError: Failed to perform, curl: (18) . See https://curl.se/libcurl/c/libcurl-errors.html first for more details.