httpx icon indicating copy to clipboard operation
httpx copied to clipboard

Identify and close single connection in AsyncClient connection pool.

Open BlackCloud37 opened this issue 4 months ago • 0 comments

Hello,

I'm trying to use httpx(in asyncio) for my web scraper program.

In my use case, for preventing my connection from banning, I usually want to limit the requests on single connection, which means:

  1. Close a connection after N requests.
  2. Close a connection if it's banned, for example, received HTTP code 429 or any invalid response.

I think both of them can be implemented from getting a connection object from a request, mapping it to it's custom state, and call aclose on it whenever needed.

But by reading the document, I cannot find any methods that I can:

  1. Identify each request's underlying connection, so I can count requests on certain connection.
  2. Close this single connection if satisfied any condition above.

So I currently implement this in "one connection per async client" way. I limited the max_connections to 1 for my client, and each time I want to drop a connection, I throw the client away and create a new one.

As the document says, it's a bad idea to do so. So is there any way I can reuse the clients for my purpose?

Any help would be much appreciated!

BlackCloud37 avatar Aug 25 '25 06:08 BlackCloud37