h2 debug assertion failed after future cancellation

[tabokie]

Version

hyper=1.5.0, h2=0.4.6

Platform

Linux runner-gijtt4lj-project-989-concurrent-0cnnp8 4.19.91-27.1.al7.x86_64 #1 SMP Tue Feb 21 11:41:32 CST 2023 x86_64 x86_64 x86_64 GNU/Linux

Description

This h2 assertion failed:

https://github.com/hyperium/h2/blob/27d12726d915e1de131de2cac5e8ee260dda8be9/src/proto/streams/prioritize.rs#L448-L52

The logs and stack trace:

2025-06-25T10:21:32.767603Z  INFO ***: receive signal(SignalKind(2)), stopping service... subscriber="replicator-v2"
2025-06-25T10:21:32.767775Z  INFO ***: takes 23.247µs to join 1 tokio tasks and 64ns to join 0 thread tasks
2025-06-25T10:21:32.767825Z  WARN huge_common::connection_manager: /root/.cargo/git/checkouts/***/connection_manager.rs:587: transport error tonic::transport::Error(Transport, hyper::Error(User(DispatchGone), "runtime dropped the dispatch task"))
2025-06-25T10:21:32.767853Z  INFO huge_client::fragments_resovler: /root/.cargo/git/checkouts/***/fragment_resolver.rs:738: Resolve fragments connection error: Status { code: Unknown, message: "transport error", source: Some(tonic::transport::Error(Transport, hyper::Error(User(DispatchGone), "runtime dropped the dispatch task"))) }, retry...
2025-06-25T10:21:32.806363Z ERROR moist::logger: /root/.cargo/git/checkouts/moist-36b3cb3c173db6d0/6dcce95/src/logger.rs:221: thread 'tokio-runtime-worker' panicked at 'state=State { inner: Closed(ScheduledLibraryReset(CANCEL)) }': /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/h2-0.4.6/src/proto/streams/prioritize.rs:443   0: <backtrace::capture::Backtrace as core::default::Default>::default
   1: moist::logger::init_logger_and_panic_hook_imp::{{closure}}
   2: std::panicking::rust_panic_with_hook
   3: std::panicking::begin_panic_handler::{{closure}}
   4: std::sys::backtrace::__rust_end_short_backtrace
   5: rust_begin_unwind
   6: core::panicking::panic_fmt
   7: h2::proto::streams::prioritize::Prioritize::try_assign_capacity
   8: h2::proto::streams::prioritize::Prioritize::poll_complete
   9: h2::proto::streams::streams::Streams<B,P>::poll_complete
  10: h2::proto::connection::Connection<T,P,B>::poll
  11: <h2::client::Connection<T,B> as core::future::future::Future>::poll
  12: <hyper::proto::h2::client::Conn<T,B> as core::future::future::Future>::poll
  13: <hyper::proto::h2::client::H2ClientFuture<B,T> as core::future::future::Future>::poll

[seanmonstar]

Interesting! Thanks for the report. Do I understand this correctly? You canceled a future (dropped it), which was probably one that was scheduled to send some data, and then shortly after the other task that tries to assign capacity blows up because the state is wrong?

[tabokie]

I took another look and it seems I was mistaken earlier. The problematic client call future is deep in our stack and was detached to the runtime. To be more specific, the client was trying to start a duplex streaming and failed at Service::call(request). Then the program was SIGINT and the entire runtime got dropped.

I currently don't have the bandwidth to repro this further, I could show some code snippets of our app here if it helps.

[tabokie]

Assuming from the tag the code is welcome? It's very noisy TBH, but here goes: https://gist.github.com/tabokie/84728599bbb475fc947a53b84d37d04e , failing at fragment_resolver.rs:L729