sled thread 'sled-io-2' panicked at 'assertion failed: prev.is

expected result Sled works reliably
actual result sometimes a request to my program takes a very long time, over 2 minutes. It finishes shortly after the panic below is written to the logs
sled version sled = { version = "0.34.6", default-features = false }
rustc version rustc 1.47.0 (18bf6b4f0 2020-10-07)
operating system cross compiled from Linux desktop 5.9.0-2-amd64 #1 SMP Debian 5.9.6-1 (2020-11-08) x86_64 GNU/Linux running on Linux server 4.14.180-178 #1 SMP PREEMPT Wed Sep 2 12:39:45 -03 2020 armv7l armv7l armv7l GNU/Linux
minimal code sample that helps to reproduce the issue I weren't able to create a small code sample yet, I will comment when I did
logs, panic messages, stack traces

Dec 09 15:15:53 koesters conduit-bin[32197]: thread 'sled-io-2' panicked at 'assertion failed: prev.is_none()', /cargo/registry/src/github.com-1ecc6299db9ec823/sled-0.34.6/src/pagecache/segment.rs:125:9
Dec 09 15:15:53 koesters conduit-bin[32197]: stack backtrace:
Dec 09 15:15:53 koesters conduit-bin[32197]:    0:   0xd2ff40 - std::backtrace_rs::backtrace::libunwind::trace::hf1bde01b3223f261
Dec 09 15:15:53 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/../../backtrace/src/backtrace/libunwind.rs:96
Dec 09 15:15:53 koesters conduit-bin[32197]:    1:   0xd2ff40 - std::backtrace_rs::backtrace::trace_unsynchronized::h42a66254cf92569f
Dec 09 15:15:53 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/../../backtrace/src/backtrace/mod.rs:66
Dec 09 15:15:53 koesters conduit-bin[32197]:    2:   0xd2ff40 - std::sys_common::backtrace::_print_fmt::h3b6ee94a5a5cba09
Dec 09 15:15:53 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/sys_common/backtrace.rs:79
Dec 09 15:15:53 koesters conduit-bin[32197]:    3:   0xd2ff40 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h00844bb1e6e9a129
Dec 09 15:15:53 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/sys_common/backtrace.rs:58
Dec 09 15:15:53 koesters conduit-bin[32197]:    4:   0xd65d14 - core::fmt::write::h90a6c5c868081726
Dec 09 15:15:53 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/core/src/fmt/mod.rs:1082
Dec 09 15:15:53 koesters conduit-bin[32197]:    5:   0xd29070 - std::io::Write::write_fmt::h98f5ffb8eb258811
Dec 09 15:15:53 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/io/mod.rs:1514
Dec 09 15:15:53 koesters conduit-bin[32197]:    6:   0xd32604 - std::sys_common::backtrace::_print::ha0fa139d86c6a543
Dec 09 15:15:53 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/sys_common/backtrace.rs:61
Dec 09 15:15:53 koesters conduit-bin[32197]:    7:   0xd32604 - std::sys_common::backtrace::print::h92d5e51547a9c181
Dec 09 15:15:53 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/sys_common/backtrace.rs:48
Dec 09 15:15:53 koesters conduit-bin[32197]:    8:   0xd32604 - std::panicking::default_hook::{{closure}}::h50bcdc10141407dd
Dec 09 15:15:53 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:200
Dec 09 15:15:53 koesters conduit-bin[32197]:    9:   0xd322d4 - std::panicking::default_hook::h339e47b571834fe0
Dec 09 15:16:07 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:219
Dec 09 15:16:07 koesters conduit-bin[32197]:   10:   0xd32cf8 - std::panicking::rust_panic_with_hook::h6f6ccf6289a90e32
Dec 09 15:16:07 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:569
Dec 09 15:16:07 koesters conduit-bin[32197]:   11:   0x83911c - std::panicking::begin_panic::{{closure}}::hac9f1c179738779a
Dec 09 15:16:07 koesters conduit-bin[32197]:   12:   0x8381bc - std::sys_common::backtrace::__rust_end_short_backtrace::hc6ea3d26012bad19
Dec 09 15:16:07 koesters conduit-bin[32197]:   13:   0x8456a0 - std::panicking::begin_panic::h67b02ed126fc954b
Dec 09 15:16:07 koesters conduit-bin[32197]:   14:   0x823610 - sled::pagecache::segment::SegmentCleaner::add_pids::heb8af2e232e8f1ea
Dec 09 15:16:07 koesters conduit-bin[32197]:   15:   0x8278a4 - sled::pagecache::segment::SegmentAccountant::possibly_clean_or_free_segment::h64138609cf8872b7
Dec 09 15:16:07 koesters conduit-bin[32197]:   16:   0x8289dc - sled::pagecache::segment::SegmentAccountant::stabilize::h168b12641b6fd40e
Dec 09 15:16:07 koesters conduit-bin[32197]:   17:   0x83528c - sled::pagecache::iobuf::IoBufs::write_to_log::h82bf58e350f39181
Dec 09 15:16:07 koesters conduit-bin[32197]:   18:   0x82a42c - core::ops::function::FnOnce::call_once{{vtable.shim}}::h99fec6124841da07
Dec 09 15:16:07 koesters conduit-bin[32197]:   19:   0x82f90c - sled::threadpool::perform_work::ha3151c7fd76394c0
Dec 09 15:16:07 koesters conduit-bin[32197]:   20:   0x8381ec - std::sys_common::backtrace::__rust_begin_short_backtrace::h00e234a7db78c64b
Dec 09 15:16:07 koesters conduit-bin[32197]:   21:   0x81cb74 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6f78dbd6bb671f55
Dec 09 15:16:07 koesters conduit-bin[32197]:   22:   0xd370b4 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h0dc9586fde86221e
Dec 09 15:16:07 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/alloc/src/boxed.rs:1042
Dec 09 15:16:55 koesters conduit-bin[32197]:   23:   0xd370b4 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::hf3ccd411016df2e7
Dec 09 15:16:55 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/alloc/src/boxed.rs:1042
Dec 09 15:16:55 koesters conduit-bin[32197]:   24:   0xd370b4 - std::sys::unix::thread::Thread::new::thread_start::hb1842b3fc789acf0
Dec 09 15:16:55 koesters conduit-bin[32197]:                        at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/sys/unix/thread.rs:87
Dec 09 15:16:55 koesters conduit-bin[32197]: ERROR - IO thread unexpectedly terminated.
Dec 09 15:16:55 koesters conduit-bin[32197]:                     please report this error at the sled github repo. Err(Any)

Dec 09 '20 14:12 timokoesters

How large is the database? Is it larger than 4gb?

Dec 20 '20 17:12 spacejam

I have 60002805 entries, totaling 6.5GB, in one tree

Dec 20 '20 19:12 timokoesters

Any update on this?

Feb 15 '21 15:02 SilentCicero

@SilentCicero there are a few factors that make this lower on the priority list for the time being:

likely 32-bit issue
involves a lot of code paths that have changed on master, so it might already be fixed, but maybe not
I've got a full plate with internal storage engine work that needs to get done before I feel comfortable releasing 1.0

If you have time to dig in, please do so.

Feb 15 '21 15:02 spacejam