electrs icon indicating copy to clipboard operation
electrs copied to clipboard
verified publisher icon romanz

Bug: Thread 8 "rocksdb:high0" received signal SIGILL, Illegal instruction.

Open fleitner opened this issue 3 years ago • 1 comments

Describe the bug Running v0.9.6 on raspberry pi 4B results in a crash due to SIGILL, Illegal instruction.

Actually, the problem comes from librocksdb, see the back trace below. That is a known problem fixed by https://github.com/facebook/rocksdb/pull/7233 which is already merged: https://github.com/facebook/rocksdb/commit/29f7bbef995bdf83098963799c66af742e95373f

However, I could not use the fixed versions because it seems electrs is using symbols that have changed.

So, if you run into this problem then get the rocksdb-6.12.6 that has the same APIs and apply the patch above. That resolved the issue for me.

To Reproduce Steps to reproduce the behavior:

  1. Configure and start electrs
  2. Wait it sync for the first time.
  3. Around 18000 blocks it will crash.

Backtrace: Thread 8 "rocksdb:high0" received signal SIGILL, Illegal instruction. [Switching to Thread 0x7fab1f9c00 (LWP 305)] 0x0000007fb921d91c in crc32c_arm64(unsigned int, unsigned char const*, unsigned int) () from /usr/lib/librocksdb.so.6.11

0x0000007fb921d91c in crc32c_arm64(unsigned int, unsigned char const*, unsigned int) () from /usr/lib/librocksdb.so.6.11 (gdb) bt #0 0x0000007fb921d91c in crc32c_arm64(unsigned int, unsigned char const*, unsigned int) () from /usr/lib/librocksdb.so.6.11 #1 0x0000007fb90f7854 in rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool) () from /usr/lib/librocksdb.so.6.11 #2 0x0000007fb90f9550 in rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool) () from /usr/lib/librocksdb.so.6.11 #3 0x0000007fb90f980c in rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool) () from /usr/lib/librocksdb.so.6.11 #4 0x0000007fb90fc634 in rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&) () from /usr/lib/librocksdb.so.6.11 #5 0x0000007fb8eff28c in rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBaserocksdb::Slice, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_deleterocksdb::FragmentedRangeTombstoneIterator >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_deleterocksdb::FragmentedRangeTombstoneIterator > > >, rocksdb::FileMetaData, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_deleterocksdb::IntTblPropCollectorFactory >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_deleterocksdb::IntTblPropCollectorFactory > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<unsigned long, std::allocator >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::IOStatus*, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long) () from /usr/lib/librocksdb.so.6.11 #6 0x0000007fb8fd726c in rocksdb::FlushJob::WriteLevel0Table() () from /usr/lib/librocksdb.so.6.11 #7 0x0000007fb8fd7fdc in rocksdb::FlushJob::Run(rocksdb::LogsWithPrepTracker*, rocksdb::FileMetaData*) () from /usr/lib/librocksdb.so.6.11 #8 0x0000007fb8f8248c in rocksdb::DBImpl::FlushMemTableToOutputFile(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, bool*, rocksdb::JobContext*, rocksdb::SuperVersionContext*, std::vector<unsigned long, std::allocator >&, unsigned long, rocksdb::SnapshotChecker*, rocksdb::LogBuffer*, rocksdb::Env::Priority) () from /usr/lib/librocksdb.so.6.11 #9 0x0000007fb8f85174 in rocksdb::DBImpl::FlushMemTablesToOutputFiles(rocksdb::autovector<rocksdb::DBImpl::BGFlushArg, 8ul> const&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::Env::Priority) () from /usr/lib/librocksdb.so.6.11 #10 0x0000007fb8f85fc4 in rocksdb::DBImpl::BackgroundFlush(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::FlushReason*, rocksdb::Env::Priority) () from /usr/lib/librocksdb.so.6.11 #11 0x0000007fb8f8b690 in rocksdb::DBImpl::BackgroundCallFlush(rocksdb::Env::Priority) () from /usr/lib/librocksdb.so.6.11 #12 0x0000007fb91776f4 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from /usr/lib/librocksdb.so.6.11 #13 0x0000007fb91778f8 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from /usr/lib/librocksdb.so.6.11 #14 0x0000007fb8828cac in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6 #15 0x0000007fb8afc628 in start_thread (arg=0x7fab1f9500) at pthread_create.c:477

fleitner avatar Apr 13 '22 21:04 fleitner

Thanks for awesome report with analysis and the fix! We vendor our fork of rocksdb so I think we may be able to patch it.

Kixunil avatar Apr 25 '22 10:04 Kixunil