incompatibility with the new version of rocksdb

[prusnak]

Somwhere between 6.23.3 and 6.25.3 the API or rocksdb has changed:

   Compiling electrs-rocksdb v0.15.0-e2
error[E0308]: mismatched types
   --> /private/tmp/nix-build-electrs-0.9.2.drv-0/electrs-0.9.2-vendor.tar.gz/electrs-rocksdb/src/db_options.rs:477:56
    |
477 |                 ffi::rocksdb_filterpolicy_create_bloom(bits_per_key)
    |                                                        ^^^^^^^^^^^^ expected `f64`, found `i32`
    |
help: you can convert an `i32` to an `f64`, producing the floating point representation of the integer
    |
477 |                 ffi::rocksdb_filterpolicy_create_bloom(bits_per_key.into())
    |                                                        ^^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
   --> /private/tmp/nix-build-electrs-0.9.2.drv-0/electrs-0.9.2-vendor.tar.gz/electrs-rocksdb/src/db_options.rs:479:61
    |
479 |                 ffi::rocksdb_filterpolicy_create_bloom_full(bits_per_key)
    |                                                             ^^^^^^^^^^^^ expected `f64`, found `i32`
    |
help: you can convert an `i32` to an `f64`, producing the floating point representation of the integer
    |
479 |                 ffi::rocksdb_filterpolicy_create_bloom_full(bits_per_key.into())
    |                                                             ^^^^^^^^^^^^^^^^^^^

[prusnak]

It seems the fractional bits_per_key was introduced quite some time ago, see https://github.com/facebook/rocksdb/commit/57f30322855b6ed38296ec0f4c6fb03b095fe0b3 (27 Nov 2019)

But for some reason the build is OK when I use rocksdb 6.23.3 and fails when I use rocksdb 6.25.3 🤔

[romanz]

Many thanks for reporting this issue!

Due to multiple compatibility issues (mostly with ARM platforms - e.g. #403), we have forked the rust-rocksdb crate (https://crates.io/crates/electrs-rocksdb), in order to "pin" the C++ RocksDB to 6.11.4 release (#473) to allow dynamic linking on Debian 11.

So indeed, it probably won't work with the latest RocksDB releases :(

[romanz]

FWIW, we have https://github.com/romanz/electrs/issues/201 :)

[dgyg]

I tried running electrs 0.9.3 with dynamic linking on Debian 11 but it crashed during indexing around block height 178000:

electrs[928370]: [2021-12-11T14:31:15.822Z INFO  electrs::chain] chain updated: tip=00000000000009eae2697a7aaf57e730b707b9f4530449c16d924d534d41f297, height=178000
electrs[928370]: [2021-12-11T14:31:15.833Z INFO  electrs::index] indexing 2000 blocks: [178001..180000]
systemd[1]: electrs.service: Main process exited, code=killed, status=4/ILL
systemd[1]: electrs.service: Failed with result 'signal'.
systemd[1]: electrs.service: Consumed 3min 30.291s CPU time.

System info:

• librocksdb.so.6.11.4 dynamically linked
• Debian 11.1
• Raspberry Pi 4 Model B 8GB

A binary with static linking however works fine :-)

[Kixunil]

@dgyg RPis (or ARM64 more specifically) have this problem. We didn't have time to figure it out yet.

[zciendor]

Due to multiple compatibility issues (mostly with ARM platforms - e.g. #403), we have forked the rust-rocksdb crate (https://crates.io/crates/electrs-rocksdb), in order to "pin" the C++ RocksDB to 6.11.4 release (#473) to allow dynamic linking on Debian 11.

Unfortunately dynamic linking on Debian 11 isn't (easily) possible for 32-bit ARMv7 (Odroid HC2), as Debian 11 doesn't provide a librocksdb-dev nor librocksdb package for armhf.

And compiling electrs 0.9.3 with statically linked librocksdb always results in the following error on Odroid (32-bit, armhf) and Debian 11, even with RUSTFLAGS="-C link-args=-latomic"

error: failed to run custom build command for `electrs-librocksdb-sys v6.11.4-e2`

Caused by:
  process didn't exit successfully: `/mnt/data/electrs/downloads/github/electrs/target/release/build/electrs-librocksdb-sys-ef28f0f5defa703b/build-script-build` (exit code: 1)
  --- stdout
  cargo:rerun-if-changed=rocksdb/
  TARGET = Some("armv7-unknown-linux-gnueabihf")
  OPT_LEVEL = Some("3")
  HOST = Some("armv7-unknown-linux-gnueabihf")
  CXX_armv7-unknown-linux-gnueabihf = None
  CXX_armv7_unknown_linux_gnueabihf = None
  HOST_CXX = None
  CXX = None
  CXXFLAGS_armv7-unknown-linux-gnueabihf = None
  CXXFLAGS_armv7_unknown_linux_gnueabihf = None
  HOST_CXXFLAGS = None
  CXXFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = None
...
error occurred: Command "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-march=armv7-a" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.8.1/fused-src/" "-I" "snappy/" "-I" "zstd/lib/" "-I" "zstd/lib/dictBuilder/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DSNAPPY=1" "-DZSTD=1" "-DNDEBUG=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/mnt/data/electrs/downloads/github/electrs/target/release/build/electrs-librocksdb-sys-55409859504e7684/out/rocksdb/db/db_impl/db_impl_open.o" "-c" "rocksdb/db/db_impl/db_impl_open.cc" with args "c++" did not execute successfully (status code exit code: 1).

[Kixunil]

@zciendor 32 bit ARM has multiple issues, I strongly recommend migrating to 64 bit. I also tried to compile rocksdb package for it but failed.

[androw]

I'm working on upgrading rocksdb in the Alpine Linux package tree but upgrading it will break electrs. I'm suggesting a patch (https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/30466/diffs?commit_id=2fdc0e96a8ea824b36a90b2d668af740e57f6684) to allow building with newer rocksdb. It's building fine on all of our arch (https://gitlab.alpinelinux.org/androw/aports/-/pipelines/111962)

Do you have any plan on supporting newer rocksdb (and maybe limiting for Debian build only) or patching is the way to go if we want to upgrade rocksdb?

[Kixunil]

@androw do I understand correctly that the only changes needed were dependencies and the code didn't change?

[prusnak]

the only changes needed were dependencies and the code didn't change?

Yes. That seems to be the case. However, by switching the dependency from forked version to upstream version, these two issues are no longer addressed:

• https://github.com/romanz/electrs/issues/403
• https://github.com/romanz/electrs/issues/469

@androw I think you can update your alpine changes to use rust-rocksdb 0.18.0 which has been released recently and you no longer have to use the unstable git revision anymore. Also you should drop this text from your version of Cargo.toml, because the upstream no longer work arounds these two issues:

  # Workaround the following issues:
  # - https://github.com/romanz/electrs/issues/403 (support building on ARM 32-bit)
  # - https://github.com/romanz/electrs/issues/469 (dynamic linking on Debian 11)

[Kixunil]

Then I think it should be possible to support range of versions using >= and < version operators.

[androw]

Yes, I'm only changing dependencies to use rust-rocksdb. I've updated the patch taking into account your remarks.

[Kixunil]

You didn't use upper bound which may cause breakage if an actually-incompatible rocksdb version is released.

[prusnak]

I think we might be able to maintain two different setups:

• one for modern distributions, which have recent versions of rocksdb that can be dynamically linked against recent upstream rust-rocksdb (0.18 atm) and compiled with recent versions of rust compilers (1.58.1 atm) - see https://github.com/prusnak/electrs/commit/50029c76ac47f58b825be405e4d527efab00e591 for the proposed changes - this can become the new master branch
• the other for debian 11, which has older version of rocksdb (that requires electrs-rocksdb fork) and older rust compiler (1.48) = what we have now in master branch - this can become the new debian branch

What do you think @romanz @Kixunil ?

PS: rust-rocksdb 0.18 cannot be compiled with rust 1.48 shipped with debian 11 at the moment

[Kixunil]

I see two quite clean options to support that:

• Backport the fixes into upstream and use that with ranged operators.
• Merge upstream into our fork as a separate version and use range on that

Vendors having newer rocksdb would just use their own Cargo.lock to switch version (or we could have it commited alongside to make it easier).

I'm strongly against having multiple live branches. I've seen it having ugly consequences in practice.

[prusnak]

• Backport the fixes into upstream and use that with ranged operators.
• Merge upstream into our fork as a separate version and use range on that

What is upstream and what is fork in these two cases? You mean rust-rocksdb vs electrs-rocksdb?

[Kixunil]

Yes, those two, in that order.