nautilus_trader
nautilus_trader copied to clipboard
Published
20 hours ago •
nautechsystems
nautechsystems
Timer is panicking when calling `set_time_alert_ns`
Bug Report
During a sandbox paper trading test, the timer panics when calling set_time_alert_ns. The timer is called by a new throttler which is created by the strategy directly.
The throttler is used to limit the number of limit orders sent to the exchange. Bybit has a limit of 10 orders per second. The strategy is creating orders on_quote_tick which sometimes creates too many orders.
The error happens immediately after when the throttler drops a message. Then, it tries to set a new timer I believe.
Expected Behavior
No panics. It is probably already useful if an exception is thrown in Python or Cython. Rust is complaining that 'interval_ns' is not positive, was 0 (haven't figured out yet how that is even possible :)
Actual Behavior
thread '<unnamed>' panicked at common/src/ffi/clock.rs:362:10:
called `Result::unwrap()` on an `Err` value: Condition failed: invalid u64 for 'interval_ns' not positive, was 0
Stack backtrace:
0: std::backtrace_rs::backtrace::libunwind::trace
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: std::backtrace_rs::backtrace::trace_unsynchronized
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: std::backtrace::Backtrace::create
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/backtrace.rs:331:13
3: anyhow::error::<impl anyhow::Error>::msg
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.83/src/error.rs:83:36
4: anyhow::__private::format_err
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.83/src/lib.rs:691:13
5: nautilus_core::correctness::check_positive_u64
at ./nautilus_trader/nautilus_core/core/src/correctness.rs:116:9
6: nautilus_common::timer::LiveTimer::new
at ./nautilus_trader/nautilus_core/common/src/timer.rs:253:9
7: <nautilus_common::clock::LiveClock as nautilus_common::clock::Clock>::set_time_alert_ns
at ./nautilus_trader/nautilus_core/common/src/clock.rs:340:25
8: live_clock_set_time_alert
at ./nautilus_trader/nautilus_core/common/src/ffi/clock.rs:360:5
9: __pyx_f_15nautilus_trader_6common_9component_9LiveClock_set_time_alert_ns
at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:40520:3
10: __pyx_f_15nautilus_trader_6common_9component_9Throttler__set_timer
at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:73668:3
11: __pyx_f_15nautilus_trader_6common_9component_9Throttler__limit_msg
at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:73406:5
12: __pyx_f_15nautilus_trader_6common_9component_9Throttler_send
at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:72867:5
13: __pyx_pf_15nautilus_trader_6common_9component_9Throttler_6send
at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:72998:3
14: __pyx_pw_15nautilus_trader_6common_9component_9Throttler_7send
at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:72976:13
15: _PyObject_Vectorcall
16: __PyEval_EvalFrameDefault
17: __PyEval_Vector
18: __pyx_f_15nautilus_trader_6common_5actor_5Actor_on_quote_tick
at ./nautilus_trader/build/optimized/nautilus_trader/common/actor.c:33156:23
19: __pyx_f_15nautilus_trader_6common_5actor_5Actor_handle_quote_tick
at ./nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62480:9
20: __pyx_pf_15nautilus_trader_6common_5actor_5Actor_174handle_quote_tick
at ./nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62763:3
21: __pyx_pw_15nautilus_trader_6common_5actor_5Actor_175handle_quote_tick
at ./nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62737:13
22: __pyx_f_15nautilus_trader_6common_9component_10MessageBus_publish_c
at ./nautilus_trader/build/optimized/nautilus_trader/common/component.c:67425:19
23: __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_quote_tick
at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58659:3
24: __pyx_pf_15nautilus_trader_4data_6engine_10DataEngine_122_handle_quote_tick
at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58794:3
25: __pyx_pw_15nautilus_trader_4data_6engine_10DataEngine_123_handle_quote_tick
at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58768:13
26: __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_quote_tick
at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58467:23
27: __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_data
at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:56583:5
28: __pyx_pf_15nautilus_trader_4data_6engine_10DataEngine_112_handle_data
at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:57015:3
29: __pyx_pw_15nautilus_trader_4data_6engine_10DataEngine_113_handle_data
at ./nautilus_trader/build/optimized/nautilus_trader/data/engine.c:56989:13
30: _PyObject_Vectorcall
31: __PyEval_EvalFrameDefault
32: _gen_send_ex2
33: _task_step
34: _task_wakeup
35: _cfunction_vectorcall_O
36: ___pyx_f_6uvloop_4loop_6Handle__run
37: ___pyx_f_6uvloop_4loop_4Loop__on_idle
38: ___pyx_f_6uvloop_4loop_6Handle__run
39: ___pyx_f_6uvloop_4loop_cb_idle_callback
40: _uv__run_idle
41: _uv_run
42: ___pyx_f_6uvloop_4loop_4Loop___run
43: ___pyx_f_6uvloop_4loop_4Loop__run
44: ___pyx_pw_6uvloop_4loop_4Loop_25run_forever
45: ___Pyx_PyObject_CallNoArg
46: ___pyx_pw_6uvloop_4loop_4Loop_45run_until_complete
47: _method_vectorcall_O
48: _PyObject_Vectorcall
49: __PyEval_EvalFrameDefault
50: _PyEval_EvalCode
51: _run_mod
52: __PyRun_SimpleFileObject
53: __PyRun_AnyFileObject
54: _Py_RunMain
55: _pymain_main
56: _Py_BytesMain
stack backtrace:
0: 0x30576ca6c - std::backtrace_rs::backtrace::libunwind::trace::hadefa2fe489b32be
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: 0x30576ca6c - std::backtrace_rs::backtrace::trace_unsynchronized::h37e8bb3c3fd9c67c
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x30576ca6c - std::sys_common::backtrace::_print_fmt::h4994a90b70e9aaa7
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
3: 0x30576ca6c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb478ebbfb46e27ce
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
4: 0x3057aba74 - core::fmt::rt::Argument::fmt::hfe2e626e52db21d5
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
5: 0x3057aba74 - core::fmt::write::he4d5fa2daff1f531
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
6: 0x305763274 - std::io::Write::write_fmt::hc5a47a68eba63d9f
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
7: 0x30576c8a0 - std::sys_common::backtrace::_print::hd47355bf70e1c2da
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
8: 0x30576c8a0 - std::sys_common::backtrace::print::h79bd952cc5812e7a
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
9: 0x30576f368 - std::panicking::default_hook::{{closure}}::h82301f6222887737
10: 0x30576f0b0 - std::panicking::default_hook::h1e49abbb3f1d7dbf
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
11: 0x30576f8c8 - std::panicking::rust_panic_with_hook::h1e70c5d905e30e9d
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:779:13
12: 0x30576f7bc - std::panicking::begin_panic_handler::{{closure}}::h399e32952efd26a4
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
13: 0x30576cef0 - std::sys_common::backtrace::__rust_end_short_backtrace::h2ab87f841a2323e7
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
14: 0x30576f544 - rust_begin_unwind
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
15: 0x305817acc - core::panicking::panic_fmt::h33e40d2a93cab78f
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
16: 0x305817f0c - core::result::unwrap_failed::h79c2cb9e4fc28c72
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1649:5
17: 0x3051ee5b0 - core::result::Result<T,E>::unwrap::h81d0ea2beeb4b9b4
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1073:23
18: 0x30496eb4c - live_clock_set_time_alert
at /Users/davidblom/repositories/market-master/nautilus_trader/nautilus_core/common/src/ffi/clock.rs:360:5
19: 0x304855fd8 - __pyx_f_15nautilus_trader_6common_9component_9LiveClock_set_time_alert_ns
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:40520:3
20: 0x30486e6ac - __pyx_f_15nautilus_trader_6common_9component_9Throttler__set_timer
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:73668:3
21: 0x30486de74 - __pyx_f_15nautilus_trader_6common_9component_9Throttler__limit_msg
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:73406:5
22: 0x30486d534 - __pyx_f_15nautilus_trader_6common_9component_9Throttler_send
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:72867:5
23: 0x30489347c - __pyx_pf_15nautilus_trader_6common_9component_9Throttler_6send
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:72998:3
24: 0x30489347c - __pyx_pw_15nautilus_trader_6common_9component_9Throttler_7send
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:72976:13
25: 0x1032998ac - _PyObject_Vectorcall
26: 0x103379d38 - __PyEval_EvalFrameDefault
27: 0x103371340 - __PyEval_Vector
28: 0x167b15b00 - __pyx_f_15nautilus_trader_6common_5actor_5Actor_on_quote_tick
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/actor.c:33156:23
29: 0x167b3bb24 - __pyx_f_15nautilus_trader_6common_5actor_5Actor_handle_quote_tick
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62480:9
30: 0x167b5d24c - __pyx_pf_15nautilus_trader_6common_5actor_5Actor_174handle_quote_tick
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62763:3
31: 0x167b5d24c - __pyx_pw_15nautilus_trader_6common_5actor_5Actor_175handle_quote_tick
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/actor.c:62737:13
32: 0x30486afb0 - __pyx_f_15nautilus_trader_6common_9component_10MessageBus_publish_c
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/common/component.c:67425:19
33: 0x17e4e8508 - __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_quote_tick
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58659:3
34: 0x17e5070ac - __pyx_pf_15nautilus_trader_4data_6engine_10DataEngine_122_handle_quote_tick
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58794:3
35: 0x17e5070ac - __pyx_pw_15nautilus_trader_4data_6engine_10DataEngine_123_handle_quote_tick
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58768:13
36: 0x17e4e8784 - __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_quote_tick
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:58467:23
37: 0x17e4e5424 - __pyx_f_15nautilus_trader_4data_6engine_10DataEngine__handle_data
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:56583:5
38: 0x17e5064cc - __pyx_pf_15nautilus_trader_4data_6engine_10DataEngine_112_handle_data
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:57015:3
39: 0x17e5064cc - __pyx_pw_15nautilus_trader_4data_6engine_10DataEngine_113_handle_data
at /Users/davidblom/repositories/market-master/nautilus_trader/build/optimized/nautilus_trader/data/engine.c:56989:13
40: 0x1032998ac - _PyObject_Vectorcall
41: 0x103379d38 - __PyEval_EvalFrameDefault
42: 0x1032b27fc - _gen_send_ex2
43: 0x103048dbc - _task_step
44: 0x10304a15c - _task_wakeup
45: 0x1032e4184 - _cfunction_vectorcall_O
46: 0x17f881d3c - ___pyx_f_6uvloop_4loop_6Handle__run
47: 0x17f86d810 - ___pyx_f_6uvloop_4loop_4Loop__on_idle
48: 0x17f881ff4 - ___pyx_f_6uvloop_4loop_6Handle__run
49: 0x17f8b343c - ___pyx_f_6uvloop_4loop_cb_idle_callback
50: 0x17f932288 - _uv__run_idle
51: 0x17f92c770 - _uv_run
52: 0x17f86e1e0 - ___pyx_f_6uvloop_4loop_4Loop___run
53: 0x17f86e474 - ___pyx_f_6uvloop_4loop_4Loop__run
54: 0x17f8bd850 - ___pyx_pw_6uvloop_4loop_4Loop_25run_forever
55: 0x17f862098 - ___Pyx_PyObject_CallNoArg
56: 0x17f8c01bc - ___pyx_pw_6uvloop_4loop_4Loop_45run_until_complete
57: 0x1032a5128 - _method_vectorcall_O
58: 0x1032998ac - _PyObject_Vectorcall
59: 0x103379d38 - __PyEval_EvalFrameDefault
60: 0x103371210 - _PyEval_EvalCode
61: 0x1033c40b8 - _run_mod
62: 0x1033c25ac - __PyRun_SimpleFileObject
63: 0x1033c2034 - __PyRun_AnyFileObject
64: 0x1033e1450 - _Py_RunMain
65: 0x1033e1800 - _pymain_main
66: 0x1033e18a0 - _Py_BytesMain
fatal runtime error: failed to initiate panic, error 5
Steps to Reproduce the Problem
- Sandbox paper trading
- Send limit orders on every quote tick
- Throttle the limit orders with a throttler to not send too many orders.
Specifications
- OS platform: MacOS
- Python version: 3.11.4
-
nautilus_traderversion: develop branch, git commit ee1c9b2d24c311a6a1676d10760188762ac16af0
My hunch is that the _delta_next method of the throttler returns a zero value or very small number. Then those microseconds have already passed by the time the Rust timer starts.
Maybe it is fine if the timer allows a small negative or zero interval? If that happens, then immediately call the callback without using a thread and mark the timer as expired? The exception should still be thrown if the interval is too negative.
Not sure if this makes sense at all. The alternative would be to use a minimum time interval in the throttler which would introduce a sometimes unneeded delay.
Thanks for all the info @davidsblom
This one shouldn't be too difficult.
Done some more testing, and sometimes also stop_time was < now_ns happens.
thread 'tokio-runtime-worker' panicked at thread 'tokio-runtime-worker' panicked at common/src/timer.rs:315:17:
stop_time was < now_ns
stack backtrace:
common/src/timer.rs:315:17:
stop_time was < now_ns
0: 0x155becaec - std::backtrace_rs::backtrace::libunwind::trace::hadefa2fe489b32be
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: 0x155becaec - std::backtrace_rs::backtrace::trace_unsynchronized::h37e8bb3c3fd9c67c
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x155becaec - std::sys_common::backtrace::_print_fmt::h4994a90b70e9aaa7
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
3: 0x155becaec - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb478ebbfb46e27ce
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
4: 0x155c2baf4 - core::fmt::rt::Argument::fmt::hfe2e626e52db21d5
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
5: 0x155c2baf4 - core::fmt::write::he4d5fa2daff1f531
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
6: 0x155be32f4 - std::io::Write::write_fmt::hc5a47a68eba63d9f
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
7: 0x155bec920 - std::sys_common::backtrace::_print::hd47355bf70e1c2da
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
8: 0x155bec920 - std::sys_common::backtrace::print::h79bd952cc5812e7a
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
9: 0x155bef3e8 - std::panicking::default_hook::{{closure}}::h82301f6222887737
10: 0x155bef130 - std::panicking::default_hook::h1e49abbb3f1d7dbf
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
11: 0x155bef948 - std::panicking::rust_panic_with_hook::h1e70c5d905e30e9d
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:779:13
12: 0x155bef810 - std::panicking::begin_panic_handler::{{closure}}::h399e32952efd26a4
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:649:13
13: 0x155becf70 - std::sys_common::backtrace::__rust_end_short_backtrace::h2ab87f841a2323e7
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
14: 0x155bef5c4 - rust_begin_unwind
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
15: 0x155c97b4c - core::panicking::panic_fmt::h33e40d2a93cab78f
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
16: 0x154e0cbdc - nautilus_common::timer::LiveTimer::start::{{closure}}::habba92d2a4238457
at /Users/davidblom/repositories/market-master/nautilus_trader/nautilus_core/common/src/timer.rs:315:17
17: 0x154d48b2c - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h0d2fa539e1e1b7f1
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
18: 0x154d48798 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h561cb9fb250495a4
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
19: 0x154d48798 - tokio::runtime::task::core::Core<T,S>::poll::h2c62a4c97c7bbdec
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
20: 0x154d80aa8 - tokio::runtime::task::harness::poll_future::{{closure}}::h0436336731ad585f
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
21: 0x154d9aeb8 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hbfb5aaa8c7b25d79
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
22: 0x154dd4634 - std::panicking::try::do_call::h1446aee4bafdd9ec
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
23: 0x154dd5010 - ___rust_try
24: 0x154dd4438 - std::panicking::try::hc607503b50ecf132
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
25: 0x154d921a8 - std::panic::catch_unwind::hb8c43dbb92c3ceef
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
26: 0x154d808e8 - tokio::runtime::task::harness::poll_future::h71b9ed50b8b47066
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
27: 0x154d80d5c - tokio::runtime::task::harness::Harness<T,S>::poll_inner::ha1ce53a7c1f58773
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
28: 0x154d81440 - tokio::runtime::task::harness::Harness<T,S>::poll::h2da67552fda7edb9
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
29: 0x154dcbfa0 - tokio::runtime::task::raw::poll::h2f6358036e21e2cf
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
30: 0x154eb04b0 - tokio::runtime::task::raw::RawTask::poll::hafbff39549565f0b
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
31: 0x154eb944c - tokio::runtime::task::LocalNotified<S>::run::h246bca158db8637b
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
32: 0x154eabfbc - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h6070f5802fb1ec23
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:576:13
33: 0x154eabea0 - tokio::runtime::coop::with_budget::h7fc1cf45bd777a8a
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
34: 0x154eabea0 - tokio::runtime::coop::budget::h2d07096d46097b77
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
35: 0x154eabea0 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h28474f49af5b4a1a
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
36: 0x154eab7ac - tokio::runtime::scheduler::multi_thread::worker::Context::run::h4b51da22208e01f3
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
37: 0x154eab53c - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::hef4f7dbccbd300b4
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
38: 0x154f16248 - tokio::runtime::context::scoped::Scoped<T>::set::h48967c3e1e3c9416
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
39: 0x154ef7318 - tokio::runtime::context::set_scheduler::{{closure}}::ha85412cf3c5d248c
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
40: 0x154e989c8 - std::thread::local::LocalKey<T>::try_with::hf6966b26ac81f482
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/local.rs:270:16
41: 0x154e95144 - std::thread::local::LocalKey<T>::with::hcf8f7976ac6e8c37
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/local.rs:246:9
42: 0x154ef725c - tokio::runtime::context::set_scheduler::h4e98e3ae7b2ac870
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:9
43: 0x154eab460 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h9ef275b241e76932
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
44: 0x154f1aa3c - tokio::runtime::context::runtime::enter_runtime::h2dad1aac06750521
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
45: 0x154eab2a0 - tokio::runtime::scheduler::multi_thread::worker::run::h0785b03457e9312c
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
46: 0x154eab14c - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::hfc6e621eac9a0a4b
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
47: 0x154ee4eec - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h9de3dca6c6d00643
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
48: 0x154e5d940 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hfec594de548332cb
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
49: 0x154e5b7ac - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hb93a0dbdba224e7a
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
50: 0x154e5b7ac - tokio::runtime::task::core::Core<T,S>::poll::hd25500e57906fe49
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
51: 0x154e50a60 - tokio::runtime::task::harness::poll_future::{{closure}}::h96d98bbba712eade
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
52: 0x154edb944 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h531849d3adde8f5f
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
53: 0x154e36c04 - std::panicking::try::do_call::h1ec80eed4e700941
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
54: 0x154e3db94 - ___rust_try
55: 0x154e3102c - std::panicking::try::h08e79c44f7072e87
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
56: 0x154efff28 - std::panic::catch_unwind::h35fa19fb2c869dfb
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
57: 0x154e4d884 - tokio::runtime::task::harness::poll_future::h739e178f11ff8fd0
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
58: 0x154e417a8 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h3721f6c38aeae65d
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
59: 0x154e409c4 - tokio::runtime::task::harness::Harness<T,S>::poll::hb8fa6cc544bf5888
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
60: 0x154eb0b00 - tokio::runtime::task::raw::poll::h82f7e00145ca6939
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
61: 0x154eb04b0 - tokio::runtime::task::raw::RawTask::poll::hafbff39549565f0b
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
62: 0x154eb9548 - tokio::runtime::task::UnownedTask<S>::run::h325b97c52029e9a1
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
63: 0x154e8c928 - tokio::runtime::blocking::pool::Task::run::haf89b44256125981
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
64: 0x154e94000 - tokio::runtime::blocking::pool::Inner::run::h861cf3bdaec96f81
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
65: 0x154e93db4 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h4d4597093fed2b6e
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
66: 0x154f25184 - std::sys_common::backtrace::__rust_begin_short_backtrace::h5e5ad456e93498fa
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:155:18
67: 0x154edae78 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h023866687c51ede6
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:529:17
68: 0x154edb814 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h46685ce984f28085
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
69: 0x154e3959c - std::panicking::try::do_call::hf5618dba7a87df90
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
70: 0x154e3db94 - ___rust_try
71: 0x154e343a0 - std::panicking::try::h9a5be4bdbb5cc6aa
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
72: 0x154edacfc - std::panic::catch_unwind::hbd10baa2f7d47eec
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
73: 0x154edacfc - std::thread::Builder::spawn_unchecked_::{{closure}}::hf82a72b81ac2d572
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:528:30
74: 0x154e6d0a8 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h30f9d5df81b8dbf0
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
75: 0x155bf7d28 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h122931d29f919766
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
76: 0x155bf7d28 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h222f385c86e71c13
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
77: 0x155bf7d28 - std::sys::unix::thread::Thread::new::thread_start::h8dda2af345bab446
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
78: 0x181b8af94 - __pthread_joiner_wake
stack backtrace:
0: 0x155becaec - std::backtrace_rs::backtrace::libunwind::trace::hadefa2fe489b32be
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: 0x155becaec - std::backtrace_rs::backtrace::trace_unsynchronized::h37e8bb3c3fd9c67c
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x155becaec - std::sys_common::backtrace::_print_fmt::h4994a90b70e9aaa7
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
3: 0x155becaec - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb478ebbfb46e27ce
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
4: 0x155c2baf4 - core::fmt::rt::Argument::fmt::hfe2e626e52db21d5
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
5: 0x155c2baf4 - core::fmt::write::he4d5fa2daff1f531
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
6: 0x155be32f4 - std::io::Write::write_fmt::hc5a47a68eba63d9f
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
7: 0x155bec920 - std::sys_common::backtrace::_print::hd47355bf70e1c2da
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
8: 0x155bec920 - std::sys_common::backtrace::print::h79bd952cc5812e7a
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
9: 0x155bef3e8 - std::panicking::default_hook::{{closure}}::h82301f6222887737
10: 0x155bef130 - std::panicking::default_hook::h1e49abbb3f1d7dbf
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
11: 0x155bef948 - std::panicking::rust_panic_with_hook::h1e70c5d905e30e9d
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:779:13
12: 0x155bef810 - std::panicking::begin_panic_handler::{{closure}}::h399e32952efd26a4
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:649:13
13: 0x155becf70 - std::sys_common::backtrace::__rust_end_short_backtrace::h2ab87f841a2323e7
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
14: 0x155bef5c4 - rust_begin_unwind
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
15: 0x155c97b4c - core::panicking::panic_fmt::h33e40d2a93cab78f
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
16: 0x154e0cbdc - nautilus_common::timer::LiveTimer::start::{{closure}}::habba92d2a4238457
at /Users/davidblom/repositories/market-master/nautilus_trader/nautilus_core/common/src/timer.rs:315:17
17: 0x154d48b2c - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h0d2fa539e1e1b7f1
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
18: 0x154d48798 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h561cb9fb250495a4
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
19: 0x154d48798 - tokio::runtime::task::core::Core<T,S>::poll::h2c62a4c97c7bbdec
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
20: 0x154d80aa8 - tokio::runtime::task::harness::poll_future::{{closure}}::h0436336731ad585f
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
21: 0x154d9aeb8 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hbfb5aaa8c7b25d79
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
22: 0x154dd4634 - std::panicking::try::do_call::h1446aee4bafdd9ec
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
23: 0x154dd5010 - ___rust_try
24: 0x154dd4438 - std::panicking::try::hc607503b50ecf132
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
25: 0x154d921a8 - std::panic::catch_unwind::hb8c43dbb92c3ceef
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
26: 0x154d808e8 - tokio::runtime::task::harness::poll_future::h71b9ed50b8b47066
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
27: 0x154d80d5c - tokio::runtime::task::harness::Harness<T,S>::poll_inner::ha1ce53a7c1f58773
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
28: 0x154d81440 - tokio::runtime::task::harness::Harness<T,S>::poll::h2da67552fda7edb9
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
29: 0x154dcbfa0 - tokio::runtime::task::raw::poll::h2f6358036e21e2cf
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
30: 0x154eb04b0 - tokio::runtime::task::raw::RawTask::poll::hafbff39549565f0b
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
31: 0x154eb944c - tokio::runtime::task::LocalNotified<S>::run::h246bca158db8637b
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
32: 0x154eabfbc - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h6070f5802fb1ec23
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:576:13
33: 0x154eabea0 - tokio::runtime::coop::with_budget::h7fc1cf45bd777a8a
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
34: 0x154eabea0 - tokio::runtime::coop::budget::h2d07096d46097b77
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
35: 0x154eabea0 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h28474f49af5b4a1a
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
36: 0x154eab7ac - tokio::runtime::scheduler::multi_thread::worker::Context::run::h4b51da22208e01f3
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
37: 0x154eab53c - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::hef4f7dbccbd300b4
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
38: 0x154f16248 - tokio::runtime::context::scoped::Scoped<T>::set::h48967c3e1e3c9416
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
39: 0x154ef7318 - tokio::runtime::context::set_scheduler::{{closure}}::ha85412cf3c5d248c
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
40: 0x154e989c8 - std::thread::local::LocalKey<T>::try_with::hf6966b26ac81f482
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/local.rs:270:16
41: 0x154e95144 - std::thread::local::LocalKey<T>::with::hcf8f7976ac6e8c37
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/local.rs:246:9
42: 0x154ef725c - tokio::runtime::context::set_scheduler::h4e98e3ae7b2ac870
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:9
43: 0x154eab460 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h9ef275b241e76932
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
44: 0x154f1aa3c - tokio::runtime::context::runtime::enter_runtime::h2dad1aac06750521
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
45: 0x154eab2a0 - tokio::runtime::scheduler::multi_thread::worker::run::h0785b03457e9312c
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
46: 0x154eab14c - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::hfc6e621eac9a0a4b
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
47: 0x154ee4eec - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h9de3dca6c6d00643
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
48: 0x154e5d940 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hfec594de548332cb
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
49: 0x154e5b7ac - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hb93a0dbdba224e7a
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
50: 0x154e5b7ac - tokio::runtime::task::core::Core<T,S>::poll::hd25500e57906fe49
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
51: 0x154e50a60 - tokio::runtime::task::harness::poll_future::{{closure}}::h96d98bbba712eade
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
52: 0x154edb944 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h531849d3adde8f5f
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
53: 0x154e36c04 - std::panicking::try::do_call::h1ec80eed4e700941
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
54: 0x154e3db94 - ___rust_try
55: 0x154e3102c - std::panicking::try::h08e79c44f7072e87
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
56: 0x154efff28 - std::panic::catch_unwind::h35fa19fb2c869dfb
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
57: 0x154e4d884 - tokio::runtime::task::harness::poll_future::h739e178f11ff8fd0
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
58: 0x154e417a8 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h3721f6c38aeae65d
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
59: 0x154e409c4 - tokio::runtime::task::harness::Harness<T,S>::poll::hb8fa6cc544bf5888
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
60: 0x154eb0b00 - tokio::runtime::task::raw::poll::h82f7e00145ca6939
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
61: 0x154eb04b0 - tokio::runtime::task::raw::RawTask::poll::hafbff39549565f0b
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
62: 0x154eb9548 - tokio::runtime::task::UnownedTask<S>::run::h325b97c52029e9a1
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
63: 0x154e8c928 - tokio::runtime::blocking::pool::Task::run::haf89b44256125981
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
64: 0x154e94000 - tokio::runtime::blocking::pool::Inner::run::h861cf3bdaec96f81
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
65: 0x154e93db4 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h4d4597093fed2b6e
at /Users/davidblom/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
66: 0x154f25184 - std::sys_common::backtrace::__rust_begin_short_backtrace::h5e5ad456e93498fa
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:155:18
67: 0x154edae78 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h023866687c51ede6
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:529:17
68: 0x154edb814 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h46685ce984f28085
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panic/unwind_safe.rs:272:9
69: 0x154e3959c - std::panicking::try::do_call::hf5618dba7a87df90
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
70: 0x154e3db94 - ___rust_try
71: 0x154e343a0 - std::panicking::try::h9a5be4bdbb5cc6aa
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
72: 0x154edacfc - std::panic::catch_unwind::hbd10baa2f7d47eec
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
73: 0x154edacfc - std::thread::Builder::spawn_unchecked_::{{closure}}::hf82a72b81ac2d572
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/thread/mod.rs:528:30
74: 0x154e6d0a8 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h30f9d5df81b8dbf0
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
75: 0x155bf7d28 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h122931d29f919766
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
76: 0x155bf7d28 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h222f385c86e71c13
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
77: 0x155bf7d28 - std::sys::unix::thread::Thread::new::thread_start::h8dda2af345bab446
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
78: 0x181b8af94 - __pthread_joiner_wake
TL;DR: I think we can fix these with a few changes to allow zero intervals and stop times in the past (which will just immediately produce a time event and expire the timer), making the live timer more robust and flexible.
For the first one, when setting an alert we still use a timer under the hood - and just set a stop time. For a very short duration it's possible for the stop time to become less than the time now. Consider this calculation where for the above scenario the interval_ns would become zero. (resulting in an immediate alert, which I think would be the expected result?)
alert_time_ns = std::cmp::max(alert_time_ns, ts_now);
let interval_ns = (alert_time_ns - ts_now).into();
Rather than this being an error condition which is very easy to run into, we can just relax the validation that the interval has to be positive (so its still possible to pass in ridiculous values such a 1, as before).
For the other error, its a similar scenario and would just result in an immediate timer expiry - which is fine. So I've also removed that condition check b7373ffea9b64f7bcaad7aeee148da7d54adf452.
One really bad error I can see by using an unsigned int for interval is if anyone passed in a negative interval_ns (which wouldn't happen with a time alert, but could happen with a timer), it's going to underflow back around to set some enormous interval the timer would never reach.
Awesome! Thanks for making the changes. Just started a new run. Will report later today how it went.
Running for 2 hours now without any warnings, errors or panics. Very nice thus far!
Still running fine. I think it is safe to assume that all issues are resolved with the sandbox.