aiohttp icon indicating copy to clipboard operation
aiohttp copied to clipboard

`ClientRequest`'s constructor makes a blocking IO call

Open cbornet opened this issue 11 months ago • 4 comments

Describe the bug

In ClientRequest's constructor there is a call to update_auth which in turn calls netrc_from_env. netrc_from_env is blocking as it reads the .netrc file from the FS. So instanciating ClientRequest in the event loop is blocking the loop.

This was detected by BlockBuster in https://github.com/aio-libs/aiohttp/pull/10433

To Reproduce

N/A

Expected behavior

I would expect ClientRequest to not block the asyncio loop.

Logs/tracebacks

N/A

Python Version

3.11

aiohttp Version

HEAD

multidict Version

6.1.0

propcache Version

0.2.1

yarl Version

1.18.3

OS

macOS

Related component

Client

Additional context

No response

Code of Conduct

  • [x] I agree to follow the aio-libs Code of Conduct

cbornet avatar Feb 08 '25 14:02 cbornet

Caching the result of netrc_from_env (eg with @functools.cache) could be a first step to minimize the impact. But in that case if the file is modified, it won't be reloaded (unless there is a blocking check of the modification time, which is still not as bad as reading the file).

cbornet avatar Feb 08 '25 14:02 cbornet

We can potentially put a proper fix into v4 and mitigations into v3, if necessary.

Dreamsorcerer avatar Feb 08 '25 19:02 Dreamsorcerer

How long is the call blocks? Do you have numbers? I expect that the filesystem cache works pretty well in this case.

asvetlov avatar Feb 08 '25 22:02 asvetlov

I don't have numbers for how long it blocks. This file is usually small, so it will probably not block for long and page cache should indeed work pretty well. I understand the tradeoff between making a breaking change and doing a small optimization. Maybe the best is to do nothing or just add some runtime cache with an os.stat check (a bit like what is done for linecache). LMK what you decide.

cbornet avatar Feb 10 '25 14:02 cbornet