curl_cffi icon indicating copy to clipboard operation
curl_cffi copied to clipboard

[BUG] Unable to make simultaneous asynchronous requests

Open zentixua opened this issue 10 months ago • 4 comments

Describe the bug Making multiple simultaneous requests to a single host results in a curl error if a TCP connection to the requested host was established earlier (a request was made previously).

To Reproduce

  1. Install httpx using pip install httpx
  2. Install curl-cffi using pip install curl-cffi
  3. Run this code:
import asyncio
import sys

from curl_cffi.requests import AsyncSession
from httpx import AsyncClient

BASE_URL = 'https://images.pexels.com'
PATHS = ['/photos/842711/pexels-photo-842711.jpeg'] * 2


async def test_httpx():
    print('----- httpx -----')
    async with AsyncClient() as session:
        await session.get(BASE_URL)
        tasks = [session.get(BASE_URL + path) for path in PATHS]
        print(await asyncio.gather(*tasks))


async def test_curl_cffi():
    print('----- curl-cffi -----')
    async with AsyncSession() as session:
        await session.get(BASE_URL)
        tasks = [session.get(BASE_URL + path) for path in PATHS]
        print(await asyncio.gather(*tasks))


if __name__ == '__main__':
    if sys.platform == 'win32':
        asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
    asyncio.run(test_httpx())
    asyncio.run(test_curl_cffi())

Expected behavior The requests will be successfully completed and the bytes of the requested images will be received with an HTTP status of 200, as occurs when using httpx.

Versions

  • OS: Windows 11
  • curl_cffi version: 0.6.2
  • pip freeze dump:
anyio==4.3.0
certifi==2024.2.2
cffi==1.16.0
curl_cffi==0.6.2
exceptiongroup==1.2.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
idna==3.7
pycparser==2.22
sniffio==1.3.1
typing_extensions==4.11.0

Additional context

  • Which session are you using? Async.
  • If using async session, which loop implementation are you using? Asyncio.

zentixua avatar Apr 18 '24 13:04 zentixua

A temporary fix is to force http/1.1, I'm still investigating a further solution.

- async with AsyncSession() as session:
+ async with AsyncSession(http_version=CurlHttpVersion.V1_1) as session:

perkfly avatar Apr 19 '24 08:04 perkfly

Hello! Are there any new details on the issue?

zentixua avatar Apr 21 '24 06:04 zentixua

Hello again, any updates?

zentixua avatar Apr 30 '24 06:04 zentixua

same error. when streaming curl_cffi.requests.errors.RequestsError: Failed to perform, curl: (18) . See https://curl.se/libcurl/c/libcurl-errors.html first for more details.

leadscloud avatar Aug 12 '24 00:08 leadscloud