risingwave
risingwave copied to clipboard
bug: compute node panicked at `minstant` time goes backward
Found this bug when benchmarking the nexmark materialized source on arm64 ec2 for 2 hours. Maybe a bug in minstant
? cc @skyzh
thread 'tokio-runtime-worker' panicked at 'supplied instant is later than self', /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:49:9
stack backtrace:
0: rust_begin_unwind
at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:142:14
2: core::panicking::panic_display
at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:72:5
3: core::panicking::panic_str
at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:56:5
4: core::option::expect_failed
at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/option.rs:1874:5
5: core::option::Option<T>::expect
at ./home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:49:9
6: minstant::instant::Instant::duration_since
at ./home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:49:9
7: <minstant::instant::Instant as core::ops::arith::Sub>::sub
at ./home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:205:9
8: minstant::instant::Instant::elapsed
at ./home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/minstant-0.1.1/src/instant.rs:120:9
9: risingwave_stream::executor::dispatch::DispatchExecutorInner::dispatch::{{closure}}
at ./ebs/risingwave/src/stream/src/executor/dispatch.rs:85:29
10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at ./rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
11: <async_stack_trace::StackTraced<F> as core::future::future::Future>::poll
12: <futures_util::future::future::fuse::Fuse<Fut> as core::future::future::Future>::poll
at ./home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/future/future/fuse.rs:86:37
13: <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}
at ./ebs/risingwave/src/stream/src/executor/dispatch.rs:272:22
I've also opened an issue on the upstream: https://github.com/tikv/minstant/issues/23
BTW, have we tested how much minstant
performs better than coarsetime
?
ministant should fall back to something similar to coarsetime on arm64, as arm64 doesn't support TSO like x86 did. It would be okay to switch to coarsetime.
Also I guess this issue happens because the future is being run on another core than Instant::now is captured.