tarpc icon indicating copy to clipboard operation
tarpc copied to clipboard

tarpc freezes/shuts off after a few days of uptime

Open TAINCER opened this issue 2 months ago • 1 comments

When tarpc is running for a longer time, after a while, each call with an existing client return the connection to the server was already shutdown, on all clients and new clients can't connect anymore, only after a server restart.

I don't know what causes this, but it generally happens after a few days of server uptime. I use this purely internally, not exposing the port beyond the machine, only to communicate from a linux host into multiple windows VM's.

This is the server impl:

async fn start_tarpc_server() -> Result<()> {
    let mut listener = tarpc::serde_transport::tcp::listen(
        (
            IpAddr::V4(Ipv4Addr::UNSPECIFIED),
            env::var("NODE_PORT")?.parse()?,
        ),
        Json::default,
    )
    .await?;
    info!("Listening on port {}", listener.local_addr().port());
    listener.config_mut().max_frame_length(usize::MAX);
    listener
        // Ignore accept errors.
        .filter_map(|r| future::ready(r.ok()))
        .map(server::BaseChannel::with_defaults)
        .map(|channel| {
            let server = SupervisorServer(channel.transport().peer_addr().unwrap());
            channel.execute(server.serve()).for_each(spawn)
        })
        .buffer_unordered(usize::MAX)
        .for_each(|_| async {})
        .await;

    Ok(())
}

I'm also trying to get more metrics/logging into this to monitor how many are connected, etc. but that's still running and I have to wait until it freezes again.

TAINCER avatar Oct 15 '25 15:10 TAINCER

Sorry to hear you're having trouble!

How many outstanding connections are there when the issue starts? At a glance, the .buffer_unordered(usize::MAX) stands out to me as potentially problematic, though I feel like it would take a ton of long lived connections to cause trouble.

tikue avatar Oct 15 '25 15:10 tikue