Migrate to RocksDB package

[rubo]

Fixes #4338 and #4349

Changes:

• The current rocksdb-sharp @ dda0e1e submodule has been replaced by RocksDB package
• Changes introduced in #4333 has been reverted

Types of changes

What types of changes does your code introduce? Put an x in the boxes that apply

• [x] Bugfix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] Documentation Update
• [ ] Code style update (formatting, renaming)
• [x] Refactoring (no functional changes, no api changes)
• [x] Build related changes
• [ ] Other (please describe):

Testing

Requires testing

• [x] Yes
• [ ] No

In case you checked yes, did you write tests??

• [ ] Yes
• [x] No

Comments about testing , should you have some (optional)

Further comments (optional)

The changes in files in Nethermind.Db.Rocks are because of switching to file scoped namespaces.

[LukaszRozmej]

Does this work on all environments? MacOS? MacOS M1? Linux ARM?

[LukaszRozmej]

One more thing - as the rocksdb version increment is not rollbackable we should add a test as a failsafe so we don't update this package by accident.

[dceleda]

Tried to run a node from this branch and I got the error that we discussed before. When I run again I got the error again but when I compiled for the second time the error disappeared.

/root/nethermind_mainnet/runtimes/linux-x64/native/librocksdb-7.4.4.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/runtimes/linux-x64/native/librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/runtimes/linux-x64/native/librocksdb-7.4.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/runtimes/linux-x64/native/librocksdb-7.4.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/runtimes/linux-x64/native/librocksdb-7.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/runtimes/linux-x64/native/librocksdb-7.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/runtimes/linux-x64/native/librocksdb.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/runtimes/linux-x64/native/librocksdb.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/native/amd64/librocksdb-7.4.4.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/native/amd64/librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/native/amd64/librocksdb-7.4.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/native/amd64/librocksdb-7.4.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/native/amd64/librocksdb-7.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/native/amd64/librocksdb-7.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/native/amd64/librocksdb.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/native/amd64/librocksdb.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/native/librocksdb-7.4.4.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/native/librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/native/librocksdb-7.4.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/native/librocksdb-7.4.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/native/librocksdb-7.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/native/librocksdb-7.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/native/librocksdb.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/native/librocksdb.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/amd64/librocksdb-7.4.4.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/amd64/librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/amd64/librocksdb-7.4.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/amd64/librocksdb-7.4.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/amd64/librocksdb-7.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/amd64/librocksdb-7.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/amd64/librocksdb.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/amd64/librocksdb.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/librocksdb-7.4.4.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/librocksdb-7.4.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/librocksdb-7.4.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/librocksdb-7.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/librocksdb-7.so: cannot open shared object file: No such file or directory
/root/nethermind_mainnet/librocksdb.so: (NativeLoadException) dlopen: /root/nethermind_mainnet/librocksdb.so: cannot open shared object file: No such file or directory
librocksdb-7.4.4.so: (NativeLoadException) dlopen: librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
librocksdb-7.4.so: (NativeLoadException) dlopen: librocksdb-7.4.so: cannot open shared object file: No such file or directory
librocksdb-7.so: (NativeLoadException) dlopen: librocksdb-7.so: cannot open shared object file: No such file or directory
librocksdb.so: (NativeLoadException) dlopen: librocksdb.so: cannot open shared object file: No such file or directory
   at NativeImport.Importers.Import[T](INativeLibImporter importer, String libName, String version, Boolean suppressUnload)
   at NativeImport.Auto.Import[T](String name, String version, Boolean suppressUnload)
   at RocksDbSharp.Native..cctor()
   --- End of inner exception stack trace ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Nethermind.Db.RocksDbInitializer.InitAllAsync() in /root/src/nethermind/src/Nethermind/Nethermind.Db/RocksDbInitializer.cs:line 81
   at Nethermind.Db.StandardDbInitializer.InitStandardDbsAsync(Boolean useReceiptsDb) in /root/src/nethermind/src/Nethermind/Nethermind.Db/StandardDbInitializer.cs:line 51
   at Nethermind.Init.Steps.InitDatabase.Execute(CancellationToken _) in /root/src/nethermind/src/Nethermind/Nethermind.Init/Steps/InitDatabase.cs:line 63
2022-08-07 23:02:28.8946|  DB memory:          1096MB
2022-08-07 23:02:29.3147|Error during ethereum runner start System.ArgumentException: bloom database has not been registered in DbProvider.
   at Nethermind.Db.DbProvider.GetDb[T](String dbName) in /root/src/nethermind/src/Nethermind/Nethermind.Db/DbProvider.cs:line 50
   at Nethermind.Db.IDbProvider.get_BloomDb() in /root/src/nethermind/src/Nethermind/Nethermind.Db/IDbProvider.cs:line 39
   at Nethermind.Init.Steps.InitializeBlockTree.Execute(CancellationToken cancellationToken) in /root/src/nethermind/src/Nethermind/Nethermind.Init/Steps/InitializeBlockTree.cs:line 54
   at Nethermind.Init.Steps.EthereumStepsManager.RunOneRoundOfInitialization(CancellationToken cancellationToken) in /root/src/nethermind/src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 140
   at Nethermind.Init.Steps.EthereumStepsManager.InitializeAll(CancellationToken cancellationToken) in /root/src/nethermind/src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 108
   at Nethermind.Runner.Ethereum.EthereumRunner.Start(CancellationToken cancellationToken) in /root/src/nethermind/src/Nethermind/Nethermind.Runner/Ethereum/EthereumRunner.cs:line 49

[rubo]

@dceleda Make sure libsnappy, libc6-dev are installed.

[dceleda]

@rubo building for the second time fixes the error. I use the pullandbuild.sh script on my VM. So I don't think it's an issue related to mentioned by you missing libraries. It could be related only to dev environments when we switch between branches and some leftovers are not deleted.

[LukaszRozmej]

Maybe its about some Targets in the csproj file? Something not getting copied?

[LukaszRozmej]

Have same issue on VM, need to rebuild it 2 times. This looks like build/copy order issue. Might be reproducable by clean and build or build from freshly cloned repository. This can also potentially break release packages

2022-08-10 15:48:19.7680|RocksDb was not found, please make sure it is installed on your machine.
 On macOs : 'brew install rocksdb' System.TypeInitializationException: The type initializer for 'RocksDbSharp.Native' threw an exception.
 ---> NativeImport.NativeLoadException: Unable to locate rocksdb native library, either install it, or use RocksDbNative nuget package
Searched:
/root/nethermind_xdai/runtimes/linux-x64/native/librocksdb-7.4.4.so: (NativeLoadException) dlopen: /root/nethermind_xdai/runtimes/linux-x64/native/librocksdb-7.4.4.so: cannot open shared object file: No such file
 or directory
/root/nethermind_xdai/runtimes/linux-x64/native/librocksdb-7.4.so: (NativeLoadException) dlopen: /root/nethermind_xdai/runtimes/linux-x64/native/librocksdb-7.4.so: cannot open shared object file: No such file or
directory
/root/nethermind_xdai/runtimes/linux-x64/native/librocksdb-7.so: (NativeLoadException) dlopen: /root/nethermind_xdai/runtimes/linux-x64/native/librocksdb-7.so: cannot open shared object file: No such file or dire
ctory
/root/nethermind_xdai/runtimes/linux-x64/native/librocksdb.so: (NativeLoadException) dlopen: /root/nethermind_xdai/runtimes/linux-x64/native/librocksdb.so: cannot open shared object file: No such file or director
y
/root/nethermind_xdai/native/amd64/librocksdb-7.4.4.so: (NativeLoadException) dlopen: /root/nethermind_xdai/native/amd64/librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/native/amd64/librocksdb-7.4.so: (NativeLoadException) dlopen: /root/nethermind_xdai/native/amd64/librocksdb-7.4.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/native/amd64/librocksdb-7.so: (NativeLoadException) dlopen: /root/nethermind_xdai/native/amd64/librocksdb-7.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/native/amd64/librocksdb.so: (NativeLoadException) dlopen: /root/nethermind_xdai/native/amd64/librocksdb.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/native/librocksdb-7.4.4.so: (NativeLoadException) dlopen: /root/nethermind_xdai/native/librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/native/librocksdb-7.4.so: (NativeLoadException) dlopen: /root/nethermind_xdai/native/librocksdb-7.4.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/native/librocksdb-7.so: (NativeLoadException) dlopen: /root/nethermind_xdai/native/librocksdb-7.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/native/librocksdb.so: (NativeLoadException) dlopen: /root/nethermind_xdai/native/librocksdb.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/amd64/librocksdb-7.4.4.so: (NativeLoadException) dlopen: /root/nethermind_xdai/amd64/librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/amd64/librocksdb-7.4.so: (NativeLoadException) dlopen: /root/nethermind_xdai/amd64/librocksdb-7.4.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/amd64/librocksdb-7.so: (NativeLoadException) dlopen: /root/nethermind_xdai/amd64/librocksdb-7.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/amd64/librocksdb.so: (NativeLoadException) dlopen: /root/nethermind_xdai/amd64/librocksdb.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/librocksdb-7.4.4.so: (NativeLoadException) dlopen: /root/nethermind_xdai/librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/librocksdb-7.4.so: (NativeLoadException) dlopen: /root/nethermind_xdai/librocksdb-7.4.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/librocksdb-7.so: (NativeLoadException) dlopen: /root/nethermind_xdai/librocksdb-7.so: cannot open shared object file: No such file or directory
/root/nethermind_xdai/librocksdb.so: (NativeLoadException) dlopen: /root/nethermind_xdai/librocksdb.so: cannot open shared object file: No such file or directory
librocksdb-7.4.4.so: (NativeLoadException) dlopen: librocksdb-7.4.4.so: cannot open shared object file: No such file or directory
librocksdb-7.4.so: (NativeLoadException) dlopen: librocksdb-7.4.so: cannot open shared object file: No such file or directory
librocksdb-7.so: (NativeLoadException) dlopen: librocksdb-7.so: cannot open shared object file: No such file or directory
librocksdb.so: (NativeLoadException) dlopen: librocksdb.so: cannot open shared object file: No such file or directory
   at NativeImport.Importers.Import[T](INativeLibImporter importer, String libName, String version, Boolean suppressUnload)
   at NativeImport.Auto.Import[T](String name, String version, Boolean suppressUnload)
   at RocksDbSharp.Native..cctor()
   --- End of inner exception stack trace ---
   at Nethermind.Db.RocksDbInitializer.CreateDb(RocksDbSettings settings) in /root/src/nethermind/src/Nethermind/Nethermind.Db/RocksDbInitializer.cs:line 60
   at Nethermind.Db.RocksDbInitializer.<>c__DisplayClass12_0.<RegisterDb>b__0() in /root/src/nethermind/src/Nethermind/Nethermind.Db/RocksDbInitializer.cs:line 51
   at RocksDbSharp.BlockBasedTableOptions..ctor()
   at Nethermind.Db.Rocks.DbOnTheRocks.BuildOptions(IDbConfig dbConfig) in /root/src/nethermind/src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 170
   at Nethermind.Db.Rocks.DbOnTheRocks.Init(String basePath, String dbPath, IDbConfig dbConfig, ILogManager logManager, ColumnFamilies columnFamilies, Boolean deleteOnStart) in /root/src/nethermind/src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 102
   at Nethermind.Db.Rocks.DbOnTheRocks..ctor(String basePath, RocksDbSettings rocksDbSettings, IDbConfig dbConfig, ILogManager logManager, ColumnFamilies columnFamilies) in /root/src/nethermind/src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 75
   at Nethermind.Db.Rocks.RocksDbFactory.CreateDb(RocksDbSettings rocksDbSettings) in /root/src/nethermind/src/Nethermind/Nethermind.Db.Rocks/RocksDbFactory.cs:line 39
   at Nethermind.Db.RocksDbInitializer.CreateDb(RocksDbSettings settings) in /root/src/nethermind/src/Nethermind/Nethermind.Db/RocksDbInitializer.cs:line 60
   at Nethermind.Db.RocksDbInitializer.<>c__DisplayClass12_0.<RegisterDb>b__0() in /root/src/nethermind/src/Nethermind/Nethermind.Db/RocksDbInitializer.cs:line 51
   at Nethermind.Db.RocksDbInitializer.<>c__DisplayClass14_0.<AddRegisterAction>b__0() in /root/src/nethermind/src/Nethermind/Nethermind.Db/RocksDbInitializer.cs:line 57
   at Nethermind.Db.RocksDbInitializer.<>c__DisplayClass18_0.<InitAllAsync>b__0() in /root/src/nethermind/src/Nethermind/Nethermind.Db/RocksDbInitializer.cs:line 78
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__271_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Nethermind.Db.RocksDbInitializer.InitAllAsync() in /root/src/nethermind/src/Nethermind/Nethermind.Db/RocksDbInitializer.cs:line 81
   at Nethermind.Db.StandardDbInitializer.InitStandardDbsAsync(Boolean useReceiptsDb) in /root/src/nethermind/src/Nethermind/Nethermind.Db/StandardDbInitializer.cs:line 51
   at Nethermind.Init.Steps.InitDatabase.Execute(CancellationToken _) in /root/src/nethermind/src/Nethermind/Nethermind.Init/Steps/InitDatabase.cs:line 63
2022-08-10 15:48:19.9111|Error during ethereum runner start System.ArgumentException: bloom database has not been registered in DbProvider.
   at Nethermind.Db.DbProvider.GetDb[T](String dbName) in /root/src/nethermind/src/Nethermind/Nethermind.Db/DbProvider.cs:line 50
   at Nethermind.Db.IDbProvider.get_BloomDb() in /root/src/nethermind/src/Nethermind/Nethermind.Db/IDbProvider.cs:line 39
   at Nethermind.Init.Steps.InitializeBlockTree.Execute(CancellationToken cancellationToken) in /root/src/nethermind/src/Nethermind/Nethermind.Init/Steps/InitializeBlockTree.cs:line 54
   at Nethermind.Init.Steps.EthereumStepsManager.RunOneRoundOfInitialization(CancellationToken cancellationToken) in /root/src/nethermind/src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 140
   at Nethermind.Init.Steps.EthereumStepsManager.InitializeAll(CancellationToken cancellationToken) in /root/src/nethermind/src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 108
   at Nethermind.Runner.Ethereum.EthereumRunner.Start(CancellationToken cancellationToken) in /root/src/nethermind/src/Nethermind/Nethermind.Runner/Ethereum/EthereumRunner.cs:line 49

[LukaszRozmej]

Build packages pipeline fails: https://github.com/NethermindEth/nethermind/runs/7770755255?check_suite_focus=true

[LukaszRozmej]

I've also built this docker: https://hub.docker.com/layers/nethermind/nethermindeth/nethermind/rocksdb-nuget/images/sha256-df271315415d863647a5f389f34f13d8f9bdaf5ac81cc3c448af1fa59e31daa2?context=explore

with this pipeline: https://github.com/NethermindEth/nethermind/runs/7771466879?check_suite_focus=true

we need to check if it works

[rubo]

• @LukaszRozmej Your Docker image works.
• Our release workflow makes working binaries.
• I wasn't able to reproduce NativeImport.NativeLoadException locally neither on Docker nor on Linux, nor on Windows.

So far, the NativeImport.NativeLoadException has been reported by @LukaszRozmej and @dceleda on some VMs using the pullandbuild.sh. It looks like a problem specific to the environment/scripts being used. Running pullandbuild.sh locally made working binaries, however, there were a lot of warnings like warning MSB3026: Could not copy "<source>.dylib" to "<destination>.dylib". Beginning retry 1 in 1000ms. The process cannot access the file '<path>.dylib' because it is being used by another process.. With this said, I presume something similar might happen on VMs causing some files to miss in the output directory and the next builds could fix this. Anyway, this should not impact our releases and hence, our end-users.

I need more info to investigate further, so let's continue the discussion internally.