hyper icon indicating copy to clipboard operation
hyper copied to clipboard

Test failure in `http2_connect_detect_close`

Open Darksonn opened this issue 10 months ago • 2 comments

The following spurious test failure has been observed in Tokio's CI setup:

failures:

---- conn::http2_connect_detect_close stdout ----

thread 'conn::http2_connect_detect_close' panicked at tests\client.rs:2377:14:
drop with 1s: Elapsed(())
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\std\src\panicking.rs:697
   1: core::panicking::panic_fmt
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\core\src\panicking.rs:75
   2: core::result::unwrap_failed
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\core\src\result.rs:1704
   3: enum2$<core::result::Result<enum2$<core::result::Result<tuple$<>,futures_channel::oneshot::Canceled> >,tokio::time::error::Elapsed> >::expect<enum2$<core::result::Result<tuple$<>,futures_channel::oneshot::Canceled> >,tokio::time::error::Elapsed>
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359\library\core\src\result.rs:1061
   4: client::conn::http2_connect_detect_close::async_block$0
             at .\tests\client.rs:2375
   5: core::future::future::impl$1::poll<ref_mut$<dyn$<core::future::future::Future<assoc$<Output,tuple$<> > > > > >
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359\library\core\src\future\future.rs:124
   6: tokio::runtime::park::impl$4::block_on::closure$0<core::pin::Pin<ref_mut$<dyn$<core::future::future::Future<assoc$<Output,tuple$<> > > > > > >
             at D:\a\tokio\tokio\tokio\src\runtime\park.rs:285
   7: tokio::task::coop::with_budget
             at D:\a\tokio\tokio\tokio\src\task\coop\mod.rs:167
   8: tokio::task::coop::budget
             at D:\a\tokio\tokio\tokio\src\task\coop\mod.rs:133
   9: tokio::runtime::park::CachedParkThread::block_on<core::pin::Pin<ref_mut$<dyn$<core::future::future::Future<assoc$<Output,tuple$<> > > > > > >
             at D:\a\tokio\tokio\tokio\src\runtime\park.rs:285
  10: tokio::runtime::context::blocking::BlockingRegionGuard::block_on<core::pin::Pin<ref_mut$<dyn$<core::future::future::Future<assoc$<Output,tuple$<> > > > > > >
             at D:\a\tokio\tokio\tokio\src\runtime\context\blocking.rs:66
  11: tokio::runtime::scheduler::multi_thread::impl$0::block_on::closure$0<core::pin::Pin<ref_mut$<dyn$<core::future::future::Future<assoc$<Output,tuple$<> > > > > > >
             at D:\a\tokio\tokio\tokio\src\runtime\scheduler\multi_thread\mod.rs:87
  12: tokio::runtime::context::runtime::enter_runtime<tokio::runtime::scheduler::multi_thread::impl$0::block_on::closure_env$0<core::pin::Pin<ref_mut$<dyn$<core::future::future::Future<assoc$<Output,tuple$<> > > > > > >,tuple$<> >
             at D:\a\tokio\tokio\tokio\src\runtime\context\runtime.rs:65
  13: tokio::runtime::scheduler::multi_thread::MultiThread::block_on<core::pin::Pin<ref_mut$<dyn$<core::future::future::Future<assoc$<Output,tuple$<> > > > > > >
             at D:\a\tokio\tokio\tokio\src\runtime\scheduler\multi_thread\mod.rs:86
  14: tokio::runtime::runtime::Runtime::block_on_inner<core::pin::Pin<ref_mut$<dyn$<core::future::future::Future<assoc$<Output,tuple$<> > > > > > >
             at D:\a\tokio\tokio\tokio\src\runtime\runtime.rs:358
  15: tokio::runtime::runtime::Runtime::block_on<core::pin::Pin<ref_mut$<dyn$<core::future::future::Future<assoc$<Output,tuple$<> > > > > > >
             at D:\a\tokio\tokio\tokio\src\runtime\runtime.rs:330
  16: client::conn::http2_connect_detect_close
             at .\tests\client.rs:2375
  17: client::conn::http2_connect_detect_close::closure$0
             at .\tests\client.rs:2306
  18: core::ops::function::FnOnce::call_once<client::conn::http2_connect_detect_close::closure_env$0,tuple$<> >
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359\library\core\src\ops\function.rs:250
  19: core::ops::function::FnOnce::call_once
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library\core\src\ops\function.rs:250
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


failures:
    conn::http2_connect_detect_close

test result: FAILED. 58 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.25s

https://github.com/tokio-rs/tokio/actions/runs/15299588060/job/43036753531

Darksonn avatar May 28 '25 12:05 Darksonn

CI job passed when re-run.

Darksonn avatar May 28 '25 14:05 Darksonn

Hm, relying on TCP close behavior in tests seems to sometimes be flaky... Might be better if we rewrote that test using tokio_test::io...

seanmonstar avatar May 28 '25 16:05 seanmonstar

It seems that disable_keep_alive_mid_request sometimes fails too.

https://github.com/tokio-rs/tokio/actions/runs/17503750269/job/49722717146

failures:

---- disable_keep_alive_mid_request stdout ----

thread '<unnamed>' panicked at tests/server.rs:1205:9:
should receive OK response, but buf: []
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
   1: core::panicking::panic_fmt
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
   2: server::disable_keep_alive_mid_request::{{closure}}::{{closure}}
             at ./tests/server.rs:1205:9
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

thread 'disable_keep_alive_mid_request' panicked at tests/server.rs:1235:18:
called `Result::unwrap()` on an `Err` value: Any { .. }
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
   1: core::panicking::panic_fmt
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
   2: core::result::unwrap_failed
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1761:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1167:23
   4: server::disable_keep_alive_mid_request::{{closure}}
             at ./tests/server.rs:1235:18
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
   7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:742:70
   8: tokio::task::coop::with_budget
             at /Users/runner/work/tokio/tokio/tokio/src/task/coop/mod.rs:167:5
   9: tokio::task::coop::budget
             at /Users/runner/work/tokio/tokio/tokio/src/task/coop/mod.rs:133:5
  10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:742:25
  11: tokio::runtime::scheduler::current_thread::Context::enter
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:432:19
  12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:741:44
  13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:829:68
  14: tokio::runtime::context::scoped::Scoped<T>::set
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/context/scoped.rs:40:9
  15: tokio::runtime::context::set_scheduler::{{closure}}
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/context.rs:176:38
  16: std::thread::local::LocalKey<T>::try_with
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:315:12
  17: std::thread::local::LocalKey<T>::with
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:279:20
  18: tokio::runtime::context::set_scheduler
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/context.rs:176:17
  19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:829:27
  20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:729:24
  21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:200:33
  22: tokio::runtime::context::runtime::enter_runtime
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/context/runtime.rs:65:16
  23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/scheduler/current_thread/mod.rs:188:9
  24: tokio::runtime::runtime::Runtime::block_on_inner
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/runtime.rs:356:52
  25: tokio::runtime::runtime::Runtime::block_on
             at /Users/runner/work/tokio/tokio/tokio/src/runtime/runtime.rs:330:18
  26: server::disable_keep_alive_mid_request
             at ./tests/server.rs:1235:26
  27: server::disable_keep_alive_mid_request::{{closure}}
             at ./tests/server.rs:1191:42
  28: core::ops::function::FnOnce::call_once
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
  29: core::ops::function::FnOnce::call_once
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

mox692 avatar Sep 06 '25 07:09 mox692

how does #3946 look? that uses tokio::io::duplex rather than tokio_io::test, i found the latter to be a bit cumbersome for these particular tests. i believe it ought to fix spurious test failures however.

if that seems like a fitting proposal, i'd be happy to port the remaining tests in that suite over in the same manner.

cratelyn avatar Sep 06 '25 20:09 cratelyn

Oh yea that works too. tokio-test gives us more exact control, but it's not really necessary. The tcp stack just seems racy on macOS, so using duplex should fix it.

seanmonstar avatar Sep 06 '25 21:09 seanmonstar

cool! i can take care of the other tests sometime tomorrow afternoon.

cratelyn avatar Sep 06 '25 21:09 cratelyn

i opened #3949 to track the disable_keep_alive_mid_request test, as that is in a separate test suite.

#3947 should address spurious failures in the test originally mentioned in this issue. i'm going to mark this as closed, and allow #3947 to track further work on the server tests.

cratelyn avatar Sep 09 '25 17:09 cratelyn