HTTP request to sequencer failed due to broken hyper connection pool

[ConjunctiveNormalForm]

trafficstars

Describe the bug

I have an ExEx that performs onchain calls conditionally on observed blocks. During testing I noticed that it'll stop sending transactions after the following occur:

2025-04-22T05:52:39.099025Z DEBUG ReqwestTransport{url=http://localhost:8545/}: hyper_util::client::legacy::pool: reuse idle connection for ("http", localhost:8545)
2025-04-22T05:52:39.099144Z DEBUG rpc::eth: forwarding raw transaction to sequencer hash=
...
2025-04-22T05:52:39.100767Z DEBUG hyper_util::client::legacy::client: client connection error: error shutting down connection
2025-04-22T05:52:39.100799Z  WARN rpc::sequencer: HTTP request to sequencer failed err=error sending request for url (https://sepolia-sequencer.unichain.org/)
2025-04-22T05:52:39.100811Z  WARN rpc::eth: Failed to forward transaction to sequencer err=error sending request for url (https://sepolia-sequencer.unichain.org/)

weirdly all subsequent calls will also silently fail. I think that hyper error message comes from here but it feels impossible to track down from there.

Right now I'm running on ExEx with a Reth fork that disables idle connection pools entirely and I haven't run into the same issue so far. Another idea I have is to rebuild the RPC client upon seeing such error, something like this:

   impl SequencerClient {
       /// Wraps the normal RPC call to auto‑reconnect once on shutdown.
       pub async fn send_with_reconnect<Params, Resp>(
           &mut self,
           method: &str,
           params: Params
       ) -> Result<Resp, SequencerClientError>
       where
           Params: RpcSend + Clone,
           Resp: RpcRecv,
       {
           // first shot
           match self.send_rpc_call(method, params.clone()).await {
               Ok(r) => return Ok(r),
               Err(SequencerClientError::TransportError(_)) => {
                   tracing::warn!("transport died — rebuilding HTTP client…");
                   // rebuild the entire SequencerClient
                   let rebuilt = SequencerClient::new(self.endpoint()).await?;
                   *self = rebuilt;
               }
               Err(e) => return Err(e),
           }
           // retry once
           self.send_rpc_call(method, params).await
       }
   }

Let me know if the issue makes sense, and/or there is known way to get around this!

Steps to reproduce

See above

Node logs

Platform(s)

No response

Container Type

Docker

What version/commit are you on?

v1.3.12

What database version are you on?

n/a

Which chain / network are you on?

Unichain Sepolia

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• [x] I agree to follow the Code of Conduct

[mattsse]

did you import reqwest in your project as well? if so how?

[ConjunctiveNormalForm]

did you import reqwest in your project as well? if so how?

I do not explicitly import reqwest -- is that an issue?

[ConjunctiveNormalForm]

did you import reqwest in your project as well? if so how?

just gently bumping on this. Am I doing something obviously dumb?

[mattsse]

can't think of any, a bit puzzled by this myself

[github-actions[bot]]

This issue is stale because it has been open for 21 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 7 days since being marked as stale.