conda
Replace `requests` with `httpx` (gradually?)
Checklist
- [x] I added a descriptive title
- [x] I searched open requests and couldn't find a duplicate
What is the idea?
Use httpx instead of requests. It has a similar API for sync IO, and will also offer async IO if we ever support that (although it has some performance issues when compared to aiohttp).
Why is this needed?
- Mostly for HTTP/2 support, which would enable us to leverage some optimizations
- And the possibility of migrating to async networking
What should happen?
Drop requests and use httpx instead
Additional Context
This has been brought up several times in the past, but AFAIK there was no issue to capture it.
Hello there,
May I suggest looking at Niquests also? It would dramatically reduce your amount of work regarding the migration.
disclaimer: I'm involved with that org.
@jaimergp,
Would it make sense to initially roll this out as "experimental"?
We did story pointing on this, which came out to 5 based on us replacing requests with niquests. @dholth provided more feedback:
In requests, we open a threadpool to parallelize. In httpx we might need to adapt to asyncio? The error handling system is also based around requests exceptions. Maybe we could write an adapter between "fetch these things in parallel with progress bars" that is independent of a thread model or http library; we could do the repodata parsing through conda-libmamba-solver independently; at first using the new library for simple "not-authenticated not-proxy not-NTLM https connection" cases.
If we do want to switch to httpx, this issue needs to be updated to perform a spike. I'll keep the refinement label on this for now until we refine the issue further.
If you want to, I'd be interested to discuss that topic and give various though into that. There's an alternative to threading and async (at some places) you may want to investigate. and, fwiw httpx isn't thread safe.
https://github.com/dholth/conda-httpx is meant to adapt conda's auth handlers to an httpx client.
