horaedb icon indicating copy to clipboard operation
horaedb copied to clipboard

panic on `dump_sampling_memtable`

Open zouxiang1993 opened this issue 1 year ago • 1 comments

Describe this problem

It seems that the Sampler suggests an empty TimeRange Vec

2023-07-08 10:56:30.867 ERRO [common_util/src/panic.rs:42] thread 'ceres-write' panicked 'Record timestamp is not in time_ranges, timestamp:Timestamp(1688784575003), time_ranges:[]' at "analytic_engine/src/instance/flush_compaction.rs:963"
   0: common_util::panic::set_panic_hook::{{closure}}
             at /home/root/workspace/RustProjects/ceresdb/common_util/src/panic.rs:41:18
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:692:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:579:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/sys_common/backtrace.rs:137:18
   4: rust_begin_unwind
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:575:5
   5: core::panicking::panic_fmt
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panicking.rs:64:14
   6: analytic_engine::instance::flush_compaction::split_record_batch_with_time_ranges
             at /home/root/workspace/RustProjects/ceresdb/analytic_engine/src/instance/flush_compaction.rs:963:13
   7: analytic_engine::instance::flush_compaction::FlushTask::dump_sampling_memtable::{{closure}}
             at /home/root/workspace/RustProjects/ceresdb/analytic_engine/src/instance/flush_compaction.rs:538:40
   8: analytic_engine::instance::flush_compaction::FlushTask::dump_memtables::{{closure}}
             at /home/root/workspace/RustProjects/ceresdb/analytic_engine/src/instance/flush_compaction.rs:362:17
      analytic_engine::instance::flush_compaction::FlushTask::run::{{closure}}
             at /home/root/workspace/RustProjects/ceresdb/analytic_engine/src/instance/flush_compaction.rs:268:13
      analytic_engine::instance::flush_compaction::Flusher::schedule_table_flush::{{closure}}::{{closure}}
             at /home/root/workspace/RustProjects/ceresdb/analytic_engine/src/instance/flush_compaction.rs:240:54
      analytic_engine::instance::serial_executor::TableFlushScheduler::flush_sequentially::{{closure}}::{{closure}}
             at /home/root/workspace/RustProjects/ceresdb/analytic_engine/src/instance/serial_executor.rs:206:38
   9: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/core.rs:223:17
      tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/loom/std/unsafe_cell.rs:14:9
      tokio::runtime::task::core::Core<T,S>::poll
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/core.rs:212:13
      tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:476:19
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panic.rs:140:14
      tokio::runtime::task::harness::poll_future
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:464:18
      tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:198:27
      tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:152:15
      tokio::runtime::task::raw::poll
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/raw.rs:255:5
  10: tokio::runtime::task::raw::RawTask::poll
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/raw.rs:200:18
      tokio::runtime::task::LocalNotified<S>::run
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/mod.rs:394:9
      tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/scheduler/multi_thread/worker.rs:464:13
      tokio::runtime::coop::with_budget
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/coop.rs:107:5
      tokio::runtime::coop::budget
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/coop.rs:73:5
      tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  11: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
      tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
      tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/macros/scoped_tls.rs:61:9
      tokio::runtime::scheduler::multi_thread::worker::run
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  12: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
      <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/blocking/task.rs:42:21
      tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/core.rs:223:17
      tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/loom/std/unsafe_cell.rs:14:9
      tokio::runtime::task::core::Core<T,S>::poll
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/core.rs:212:13
      tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:476:19
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panic.rs:140:14
      tokio::runtime::task::harness::poll_future
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:464:18
      tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:198:27
      tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:152:15
      tokio::runtime::task::raw::poll
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/raw.rs:255:5
  13: tokio::runtime::task::raw::RawTask::poll
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/raw.rs:200:18
      tokio::runtime::task::UnownedTask<S>::run
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/mod.rs:431:9
      tokio::runtime::blocking::pool::Task::run
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/blocking/pool.rs:159:9
      tokio::runtime::blocking::pool::Inner::run
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/blocking/pool.rs:513:17
      tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/blocking/pool.rs:471:13
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/sys_common/backtrace.rs:121:18
  14: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/thread/mod.rs:558:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panic.rs:140:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/thread/mod.rs:557:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/ops/function.rs:250:5
  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/sys/unix/thread.rs:108:17
  16: start_thread
  17: clone

Server version

CeresDB Server Version: 1.2.2 Git commit: c6c306d Git branch: main Opt level: 3 Rustc version: 1.69.0-nightly Target: x86_64-unknown-linux-gnu Build date: 2023-07-07T02:25:17.842532518Z

Steps to reproduce

  1. Create table without segment_duration
  2. write some data

Expected behavior

Don't panic

Additional Information

zouxiang1993 avatar Jul 08 '23 03:07 zouxiang1993

This is the first time this error arise, I'm curious what is your data range? does this panic always happen?

jiacai2050 avatar Jul 11 '23 02:07 jiacai2050