romanz
Bug: Segmentation Fault Error at indexing blocks [180001..182000]
Describe the bug Everytime I start electrs I get a seg fault error at indexing the blocks. Specifically at blocks [180001..182000]. It's always in the same spot. Any help would be appreciated.
Electrs version v.0.10.5
To Reproduce Steps to reproduce the behavior:
- Configure and start electrs
- Wait
- See error
Expected behavior Indexing throws no errors.
Configuration
electrs.toml
Didn't find it in the directory.[2024-07-12T16:42:00.610Z INFO electrs::index] indexing 2000 blocks: [180001..182000]
Segmentation fault (core dumped)
running with gdb:
Thread 8 "rocksdb:high" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff58006c0 (LWP 3120)]
0x0000555555ba5775 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()
Environment variables: ELECTRS_X=Y;...
Arguments: --foo
System running electrs
- Deployment method: manual from source
- OS name and version: Ubuntu Server 24.04 LTS
Electrum client Client name (if not upstream desktop Electrum) and version: no electrum installed
Additional context n/a
error running with gdb:
[2024-07-12T16:55:11.385Z INFO electrs::index] indexing 2000 blocks: [180001..182000]
Thread 8 "rocksdb:high" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff58006c0 (LWP 3120)] 0x0000555555ba5775 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()
That's really weird, probably has something to do with C++ compilation. Makes me rethink my previous claim that maybe pure Rust impl isn't that interesting.
When I try to link dynamically i get these errors:
error[E0425]: cannot find function `rocksdb_block_based_options_set_block_cache_compressed` in crate `ffi`
--> /home/electrs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/electrs-rocksdb-0.19.0-e3/src/db_options.rs:532:18
|
532 | ffi::rocksdb_block_based_options_set_block_cache_compressed(self.inner, cache);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `rocksdb_block_based_options_set_block_cache`
|
::: /home/electrs/electrs/target/release/build/electrs-librocksdb-sys-9f28095edbfce9a7/out/bindings.rs:3:59227
|
3 | ... { pub fn rocksdb_block_based_options_set_block_cache (options : * mut rocksdb_block_based_table_options_t , block_cache : * mut rocksdb_cache_t) ; ...
| ---------------------------------------------------------------------------------------------------------------------------------------------- similarly named function `rocksdb_block_based_options_set_block_cache` defined here
error[E0425]: cannot find function `rocksdb_block_based_options_set_block_cache_compressed` in crate `ffi`
--> /home/electrs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/electrs-rocksdb-0.19.0-e3/src/db_options.rs:553:18
|
553 | ffi::rocksdb_block_based_options_set_block_cache_compressed(self.inner, cache.0.inner);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `rocksdb_block_based_options_set_block_cache`
|
::: /home/electrs/electrs/target/release/build/electrs-librocksdb-sys-9f28095edbfce9a7/out/bindings.rs:3:59227
|
3 | ... { pub fn rocksdb_block_based_options_set_block_cache (options : * mut rocksdb_block_based_table_options_t , block_cache : * mut rocksdb_cache_t) ; ...
| ---------------------------------------------------------------------------------------------------------------------------------------------- similarly named function `rocksdb_block_based_options_set_block_cache` defined here
error[E0061]: this function takes 9 arguments but 5 arguments were supplied
--> /home/electrs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/electrs-rocksdb-0.19.0-e3/src/db_options.rs:2525:13
|
2525 | ffi::rocksdb_options_set_plain_table_factory(
| _____________^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^-
2526 | | self.inner,
2527 | | options.user_key_length,
2528 | | options.bloom_bits_per_key,
2529 | | options.hash_table_ratio,
2530 | | options.index_sparseness,
2531 | | );
| |_____________- multiple arguments are missing
|
note: function defined here
--> /home/electrs/electrs/target/release/build/electrs-librocksdb-sys-9f28095edbfce9a7/out/bindings.rs:3:88313
|
3 | ...size) ; } extern "C" { pub fn rocksdb_options_set_plain_table_factory (arg1 : * mut rocksdb_options...
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: provide the arguments
|
2525 | ffi::rocksdb_options_set_plain_table_factory(self.inner, options.user_key_length, options.bloom_bits_per_key, options.hash_table_ratio, options.index_sparseness, /* usize */, /* i8 */, /* u8 */, /* u8 */);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Some errors have detailed explanations: E0061, E0425.
For more information about an error, try `rustc --explain E0061`.
error: could not compile `electrs-rocksdb` (lib) due to 3 previous errors
Yes, Ubuntu has a different rocksdb version than Debian. I wish they didn't break the API so much.
I don't have many ideas either. Maybe try switching compiler to clang? (If you used g++, or vice-versa).
@adriankae Does the error reproduce if the index DB is rebuilt from scratch?
I have the exact same issue. For me it did not help to rebuild the DB from scratch.
Found the issue for me, I was missing the build-essential package. After installing it and rebuilding electrs it works like a charm.
Is this a rocksdb issue? I think somewhere a compile time error should be introduced.
Confirming the same issue (at same block height) and the same fix build-essential was not installed.
OS: Ubuntu 24.04.1 LTS x86_64 Kernel: 6.8.0-41-generic
Just confirming as well that I was seeing the same issue at the same block height on Ubuntu 24.04. Installed "build-essential" and rebuilt electrs (static linking). After that I was able to run it and get past the previously failing block height point.
I don't think we need more reports, but new people coming to the issue can just thumbs up this comment: https://github.com/romanz/electrs/issues/1061#issuecomment-2308759609
What would be really helpful is to abort compilation if this is happening but that requires some serious digging and will likely need an issue filed against the cc crate (but only once it's confirmed to belong there - please don't pester them about it now).
If someone figures out more information about this that would explain why this happens that'd be great. Sadly I don't have that much time for this kind of thing lately.
On Ubuntu 24.04 installing clang results in Clang 18:
$ c++ --version
Ubuntu clang version 18.1.3 (1ubuntu1)
...
If I then install build-essential, GCC 13 becomes the default:
$ c++ --version
c++ (Ubuntu 13.2.0-23ubuntu4) 13.2.0
...
Also, https://github.com/mempool/electrs/pull/106 also stopped using Clang 18.
Reproduced using a Ubuntu 24.04 Docker container:
$ gdb --args /usr/bin/electrs --log-filters INFO
(gdb) r
...
[2024-10-29T14:15:03.607Z INFO electrs::chain] chain updated: tip=00000000000004ff83b6c10460b239ef4a6aa320e5fffd6c7bcedefa8c78593c, height=180000
[2024-10-29T14:15:03.608Z INFO electrs::index] indexing 2000 blocks: [180001..182000]
Thread 10 "rocksdb:high" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7bb3c1e006c0 (LWP 1032)]
0x000064b8f5f80795 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()
(gdb) bt
#0 0x000064b8f5f80795 in rocksdb::CompressionContextCache::Rep::GetZSTDUncompressData() ()
#1 0x000064b8f5f8074c in rocksdb::CompressionContextCache::GetCachedZSTDUncompressData() ()
#2 0x000064b8f5f177c7 in rocksdb::UncompressionContext::UncompressionContext(rocksdb::CompressionType) ()
#3 0x000064b8f5f4c054 in rocksdb::BlockFetcher::ReadBlockContents() ()
#4 0x000064b8f5f1f413 in rocksdb::Status rocksdb::(anonymous namespace)::ReadBlockFromFile<rocksdb::Block>(rocksdb::RandomAccessFileReader*, rocksdb::FilePrefetchBuffer*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, std::unique_ptr<rocksdb::Block, std::default_delete<rocksdb::Block> >*, rocksdb::ImmutableOptions const&, bool, bool, rocksdb::BlockType, rocksdb::UncompressionDict const&, rocksdb::PersistentCacheOptions const&, unsigned long, rocksdb::MemoryAllocator*, bool, bool, rocksdb::FilterPolicy const*, bool) ()
#5 0x000064b8f5f164a5 in rocksdb::Status rocksdb::BlockBasedTable::RetrieveBlock<rocksdb::Block>(rocksdb::FilePrefetchBuffer*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::UncompressionDict const&, rocksdb::CachableEntry<rocksdb::Block>*, rocksdb::BlockType, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, bool, bool, bool, bool) const ()
#6 0x000064b8f5f431f3 in rocksdb::BlockBasedTable::IndexReaderCommon::ReadIndexBlock(rocksdb::BlockBasedTable const*, rocksdb::FilePrefetchBuffer*, rocksdb::ReadOptions const&, bool, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, rocksdb::CachableEntry<rocksdb::Block>*) ()
#7 0x000064b8f6035b8e in rocksdb::BinarySearchIndexReader::Create(rocksdb::BlockBasedTable const*, rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, bool, bool, bool, rocksdb::BlockCacheLookupContext*, std::unique_ptr<rocksdb::BlockBasedTable::IndexReader, std::default_delete<rocksdb::BlockBasedTable::IndexReader> >*) ()
#8 0x000064b8f5f25d54 in rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, bool, bool, bool, rocksdb::BlockCacheLookupContext*, std::unique_ptr<rocksdb::BlockBasedTable::IndexReader, std::default_delete<rocksdb::BlockBasedTable::IndexReader> >*) ()
#9 0x000064b8f5f250b6 in rocksdb::BlockBasedTable::PrefetchIndexAndFilterBlocks(rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, rocksdb::BlockBasedTable*, bool, rocksdb::BlockBasedTableOptions const&, int, unsigned long, unsigned long, rocksdb::BlockCacheLookupContext*) ()
#10 0x000064b8f5f22706 in rocksdb::BlockBasedTable::Open(rocksdb::ReadOptions const&, rocksdb::ImmutableOptions const&, rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&, rocksdb::InternalKeyComparator const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, std::shared_ptr<rocksdb::CacheReservationManager>, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, int, bool, unsigned long, bool, rocksdb::TailPrefetchStats*, rocksdb::BlockCacheTracer*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::array<unsigned long, 2ul>) ()
#11 0x000064b8f5f0ca46 in rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::ReadOptions const&, rocksdb::TableReaderOptions const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, bool) const ()
#12 0x000064b8f5e2c61c in rocksdb::TableCache::GetTableReader(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, bool, bool, rocksdb::HistogramImpl*, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, int, bool, unsigned long, rocksdb::Temperature) ()
#13 0x000064b8f5e2b497 in rocksdb::TableCache::FindTable(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::Cache::Handle**, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, rocksdb::HistogramImpl*, bool, int, bool, unsigned long, rocksdb::Temperature) ()
#14 0x000064b8f5e2cb10 in rocksdb::TableCache::NewIterator(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::RangeDelAggregator*, std::shared_ptr<rocksdb::SliceTransform const> const&, rocksdb::TableReader**, rocksdb::HistogramImpl*, rocksdb::TableReaderCaller, rocksdb::Arena*, bool, int, unsigned long, rocksdb::InternalKey const*, rocksdb::InternalKey const*, bool, rocksdb::TruncatedRangeDelIterator**) ()
#15 0x000064b8f5fbc8a8 in rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned long*, unsigned long*, unsigned long*) ()
#16 0x000064b8f5de44ab in rocksdb::FlushJob::WriteLevel0Table() ()
#17 0x000064b8f5de040d in rocksdb::FlushJob::Run(rocksdb::LogsWithPrepTracker*, rocksdb::FileMetaData*, bool*) ()
#18 0x000064b8f5d71fe7 in rocksdb::DBImpl::FlushMemTableToOutputFile(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, bool*, rocksdb::JobContext*, rocksdb::SuperVersionContext*, std::vector<unsigned long, std::allocator<unsigned long> >&, unsigned long, rocksdb::SnapshotChecker*, rocksdb::LogBuffer*, rocksdb::Env::Priority) ()
#19 0x000064b8f5d73238 in rocksdb::DBImpl::FlushMemTablesToOutputFiles(rocksdb::autovector<rocksdb::DBImpl::BGFlushArg, 8ul> const&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::Env::Priority) ()
#20 0x000064b8f5d85428 in rocksdb::DBImpl::BackgroundFlush(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::FlushReason*, rocksdb::Env::Priority) ()
#21 0x000064b8f5d83b88 in rocksdb::DBImpl::BackgroundCallFlush(rocksdb::Env::Priority) ()
#22 0x000064b8f5f8e4f5 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) ()
#23 0x000064b8f5f8e759 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) ()
#24 0x00007bb3c488fbb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#25 0x00007bb3c4519a94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#26 0x00007bb3c45a6a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
with the following diff:
diff --git a/Dockerfile b/Dockerfile
index c8ec506e..a6f1dba8 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -2,19 +2,18 @@
# The maintainers of electrs are not deeply familiar with Docker, so you should DYOR.
# If you are not familiar with Docker either it's probably be safer to NOT use it.
-FROM debian:bookworm-slim AS base
+FROM ubuntu:24.04 AS base
RUN apt-get update -qqy
-RUN apt-get install -qqy librocksdb-dev curl
+RUN apt-get install -qqy curl gdb bash
### Electrum Rust Server ###
FROM base AS electrs-build
RUN apt-get install -qqy cargo clang cmake
# Install electrs
WORKDIR /build/electrs
COPY . .
-ENV ROCKSDB_INCLUDE_DIR=/usr/include
-ENV ROCKSDB_LIB_DIR=/usr/lib
RUN cargo install --locked --path .
FROM base AS result
I can confirm that on FreeBSD, installing llvm17 and recompiling electrs with the following set of variables does indeed fix this.
CC=/usr/local/bin/clang17
CXX=/usr/local/bin/clang++17
CPP=/usr/local/bin/clang-cpp17
RUSTFLAGS="-C linker=clang17"
I got similar error (electrs built with Clang):
Feb 22 16:57:42 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:42.945Z INFO electrs::chain] chain updated: tip=0000000067e871308ed838eaf7365ab1d61b8a0187f39e5457096596d73adad4, height=42000
Feb 22 16:57:42 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:42.956Z INFO electrs::index] indexing 2000 blocks: [42001..44000]
Feb 22 16:57:43 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:43.689Z INFO electrs::chain] chain updated: tip=000000007c45abdd246fb171d826b4adec60ced93208185474935445e15c9690, height=44000
Feb 22 16:57:43 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:43.694Z INFO electrs::index] indexing 2000 blocks: [44001..46000]
Feb 22 16:57:44 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:44.444Z INFO electrs::chain] chain updated: tip=0000000015222c1380636cd645fc38be3c90fab4e24d5b9be0cd31f4ecdd734b, height=46000
Feb 22 16:57:44 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:44.448Z INFO electrs::index] indexing 2000 blocks: [46001..48000]
Feb 22 16:57:45 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:45.189Z INFO electrs::chain] chain updated: tip=0000000014d177c53bc3266605968eddafc78c9d001ca43a6cec8be7d30dcdca, height=48000
Feb 22 16:57:45 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:45.197Z INFO electrs::index] indexing 2000 blocks: [48001..50000]
Feb 22 16:57:45 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:45.789Z INFO electrs::chain] chain updated: tip=00000000e2c8c94ba126169a88997233f07a9769e2b009fb10cad0e893eff2cb, height=50000
Feb 22 16:57:45 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:45.794Z INFO electrs::index] indexing 2000 blocks: [50001..52000]
Feb 22 16:57:53 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:53.256Z INFO electrs::chain] chain updated: tip=000000000057e4a39b583f13d2bc93c12f1f58d9ed432e298fc699ccb939fb13, height=52000
Feb 22 16:57:53 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:57:53.267Z INFO electrs::index] indexing 2000 blocks: [52001..54000]
Feb 22 16:58:00 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:58:00.239Z INFO electrs::chain] chain updated: tip=000000000057685c85d53ab5a10bd87c8ef9a3d07534cff3ef6f4c27bd2d718a, height=54000
Feb 22 16:58:00 qemux86-64-12-34-02 electrs[1139]: [2025-02-22T16:58:00.244Z INFO electrs::index] indexing 2000 blocks: [54001..56000]
Feb 22 16:58:03 qemux86-64-12-34-02 kernel: rocksdb:high[1146]: segfault at 0 ip 000055acd61f2dd5 sp 00007fd2e0fc0900 error 6 in electrs[55acd5bcd000+7c7000] likely on CPU 3 (core 3, socket 0)
Feb 22 16:58:03 qemux86-64-12-34-02 kernel: Code: e8 a0 57 f8 ff 85 c0 78 7a 41 8b 4e 08 41 bf ff ff ff ff 41 d3 e7 41 f7 d7 41 21 c7 49 8b 0e 4c 89 fa 48 c1 e2 06 4c 8d 34 11 <48> c7 03 00 00 00 00 48 c7 43 08 ff ff ff ff 31 f6 4c 89 f0 f0 48
It hit always the same blocks [54001..56000]
@embetrix many thanks! Could you please share the details of your system and build environment?
@romanz I built using OpenEmbedded/Yocto not quiet standard build system:
https://github.com/embetrix/meta-satobox/blob/scarthgap/recipes-extended/electrs/electrs_0.10.9.bb
I just noticed, that I do get error now only for x86_64 , aarch64 seems to be working fine