terminate called after throwing an instance of 'std::bad_alloc'

[whysthatso]

Describe the bug nethermind crashes during startup or shortly after, exact point varies.

Desktop (please complete the following information):

• OS: ETHonArm based on ubuntu 20.04, running on a rasppi 4 / 8gb
• version:

nethermind --version
2022-09-03 18-18-28.9328|Nethermind starting initialization.
2022-09-03 18-18-29.0748|Loading embedded plugins
2022-09-03 18-18-29.0755|  Found plugin type Nethermind.Consensus.AuRa.AuRaPlugin
2022-09-03 18-18-29.0756|  Found plugin type Nethermind.Consensus.Clique.CliquePlugin
2022-09-03 18-18-29.0756|  Found plugin type Nethermind.Consensus.Ethash.EthashPlugin
2022-09-03 18-18-29.0756|  Found plugin type Nethermind.Consensus.Ethash.NethDevPlugin
2022-09-03 18-18-29.0757|  Found plugin type Nethermind.Hive.HivePlugin
Resolved executing directory as /opt/nethermind.
2022-09-03 18-18-29.1154|Loading 9 assemblies from /opt/nethermind/plugins
2022-09-03 18-18-29.1164|Loading assembly Nethermind.Merge.Plugin
2022-09-03 18-18-29.1320|  Found plugin type Nethermind.Merge.Plugin
2022-09-03 18-18-29.1322|Loading assembly Nethermind.Mev
2022-09-03 18-18-29.1415|  Found plugin type Nethermind.Mev
2022-09-03 18-18-29.1416|Loading assembly Nethermind.Consensus.AuRa
2022-09-03 18-18-29.1520|Loading assembly Nethermind.Api
2022-09-03 18-18-29.1540|Loading assembly Nethermind.Merge.AuRa
2022-09-03 18-18-29.1553|  Found plugin type Nethermind.Merge.AuRa
2022-09-03 18-18-29.1554|Loading assembly Nethermind.Init
2022-09-03 18-18-29.1572|Loading assembly Nethermind.AccountAbstraction
2022-09-03 18-18-29.1652|  Found plugin type Nethermind.AccountAbstraction
2022-09-03 18-18-29.1654|Loading assembly Nethermind.HealthChecks
2022-09-03 18-18-29.1681|  Found plugin type Nethermind.HealthChecks
2022-09-03 18-18-29.1682|Loading assembly Nethermind.EthStats
2022-09-03 18-18-29.1700|  Found plugin type Nethermind.EthStats

Nethermind/v1.14.0-0-60f920b90-20220903/Arm64-Linux/6.0.8

Additional context

• data directory lives on an additionally mounted 2tb ssd
• arguments to nethermind:

  --config mainnet  --loggerConfigSource /opt/nethermind/NLog.config -dd /home/ethereum/.nethermind --Sync.SnapSync true --Sync.AncientReceiptsBarrier 11052984 --Sync.AncientBodiesBarrier 11052984 --JsonRpc.JwtSecretFile /secrets/jwtsecret --Metrics.Enabled true
  

[whysthatso]

two examples:

2022-09-03 18-22-48.5531|Nethermind starting initialization.
2022-09-03 18-22-48.6960|Loading embedded plugins
2022-09-03 18-22-48.6967|  Found plugin type Nethermind.Consensus.AuRa.AuRaPlugin
2022-09-03 18-22-48.6968|  Found plugin type Nethermind.Consensus.Clique.CliquePlugin
2022-09-03 18-22-48.6969|  Found plugin type Nethermind.Consensus.Ethash.EthashPlugin
2022-09-03 18-22-48.6969|  Found plugin type Nethermind.Consensus.Ethash.NethDevPlugin
2022-09-03 18-22-48.6969|  Found plugin type Nethermind.Hive.HivePlugin
Resolved executing directory as /opt/nethermind.
2022-09-03 18-22-48.7369|Loading 9 assemblies from /opt/nethermind/plugins
2022-09-03 18-22-48.7379|Loading assembly Nethermind.Merge.Plugin
2022-09-03 18-22-48.7536|  Found plugin type Nethermind.Merge.Plugin
2022-09-03 18-22-48.7538|Loading assembly Nethermind.Mev
2022-09-03 18-22-48.7632|  Found plugin type Nethermind.Mev
2022-09-03 18-22-48.7634|Loading assembly Nethermind.Consensus.AuRa
2022-09-03 18-22-48.7740|Loading assembly Nethermind.Api
2022-09-03 18-22-48.7760|Loading assembly Nethermind.Merge.AuRa
2022-09-03 18-22-48.7775|  Found plugin type Nethermind.Merge.AuRa
2022-09-03 18-22-48.7776|Loading assembly Nethermind.Init
2022-09-03 18-22-48.7794|Loading assembly Nethermind.AccountAbstraction
2022-09-03 18-22-48.7875|  Found plugin type Nethermind.AccountAbstraction
2022-09-03 18-22-48.7877|Loading assembly Nethermind.HealthChecks
2022-09-03 18-22-48.7905|  Found plugin type Nethermind.HealthChecks
2022-09-03 18-22-48.7906|Loading assembly Nethermind.EthStats
2022-09-03 18-22-48.7923|  Found plugin type Nethermind.EthStats
2022-09-03 18-22-49.0646|Loading NLog configuration file from /opt/nethermind/NLog.config.
2022-09-03 18-22-49.7788|Reading config file from /opt/nethermind/configs/mainnet.cfg
2022-09-03 18-22-50.6701|Configuration initialized.
2022-09-03 18-22-50.6828|Setting BaseDbPath to: /home/ethereum/.nethermind/nethermind_db/mainnet, from: nethermind_db/mainnet
2022-09-03 18-22-50.6829|Setting KeyStoreDirectory to: /home/ethereum/.nethermind/keystore, from: keystore
2022-09-03 18-22-50.6830|Setting LogDirectory to: /home/ethereum/.nethermind/logs, from: logs
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

2022-09-03 18-22-58.6330|Nethermind starting initialization.
2022-09-03 18-22-58.7745|Loading embedded plugins
2022-09-03 18-22-58.7752|  Found plugin type Nethermind.Consensus.AuRa.AuRaPlugin
2022-09-03 18-22-58.7753|  Found plugin type Nethermind.Consensus.Clique.CliquePlugin
2022-09-03 18-22-58.7753|  Found plugin type Nethermind.Consensus.Ethash.EthashPlugin
2022-09-03 18-22-58.7753|  Found plugin type Nethermind.Consensus.Ethash.NethDevPlugin
2022-09-03 18-22-58.7754|  Found plugin type Nethermind.Hive.HivePlugin
Resolved executing directory as /opt/nethermind.
2022-09-03 18-22-58.8152|Loading 9 assemblies from /opt/nethermind/plugins
2022-09-03 18-22-58.8162|Loading assembly Nethermind.Merge.Plugin
2022-09-03 18-22-58.8316|  Found plugin type Nethermind.Merge.Plugin
2022-09-03 18-22-58.8318|Loading assembly Nethermind.Mev
2022-09-03 18-22-58.8410|  Found plugin type Nethermind.Mev
2022-09-03 18-22-58.8411|Loading assembly Nethermind.Consensus.AuRa
2022-09-03 18-22-58.8516|Loading assembly Nethermind.Api
2022-09-03 18-22-58.8536|Loading assembly Nethermind.Merge.AuRa
2022-09-03 18-22-58.8549|  Found plugin type Nethermind.Merge.AuRa
2022-09-03 18-22-58.8550|Loading assembly Nethermind.Init
2022-09-03 18-22-58.8568|Loading assembly Nethermind.AccountAbstraction
2022-09-03 18-22-58.8711|  Found plugin type Nethermind.AccountAbstraction
2022-09-03 18-22-58.8714|Loading assembly Nethermind.HealthChecks
2022-09-03 18-22-58.8778|  Found plugin type Nethermind.HealthChecks
2022-09-03 18-22-58.8780|Loading assembly Nethermind.EthStats
2022-09-03 18-22-58.8822|  Found plugin type Nethermind.EthStats
2022-09-03 18-22-59.5985|Loading NLog configuration file from /opt/nethermind/NLog.config.
2022-09-03 18-23-01.3369|Reading config file from /opt/nethermind/configs/mainnet.cfg
2022-09-03 18-23-02.6181|Configuration initialized.
2022-09-03 18-23-02.6310|Setting BaseDbPath to: /home/ethereum/.nethermind/nethermind_db/mainnet, from: nethermind_db/mainnet
2022-09-03 18-23-02.6311|Setting KeyStoreDirectory to: /home/ethereum/.nethermind/keystore, from: keystore
2022-09-03 18-23-02.6311|Setting LogDirectory to: /home/ethereum/.nethermind/logs, from: logs
2022-09-03 18:23:05.4959|Using http://ipv4.icanhazip.com to get external ip 
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

[diglos]

Hi.

This happens as well on Odroid M1 and Rock 5B board ARM boards. Checked with the last version 1.14.1. Is there any way we can get a hint on what is going on? (meaning how to trace this).

Thanks!

[dannywillems]

Having the same issue on rasp 8GB

[FalcoXYZ]

@andreicupaciu no update yet. We are getting a Raspberry Pi to test ETHonArm on so we can try to reproduce this issue. Can you tell me which hardware you're getting this issue on?

[andreicupaciu]

Hi,

We just figured it out. In our case, on the used system, there were 2 packages installed: librocksdb-dev (5.17.2-3) librocksdb5.17 (5.17.2-3)

After removing them, everything seams ok.

[FalcoXYZ]

Did you receive the exact same error "terminate called after throwing an instance of 'std::bad_alloc'" when you had those packages installed? @andreicupaciu

[andreicupaciu]

@FalcoXYZ exactly the same error, yes. PS: system is x86_64, Ubuntu 20.04, 5.4.0-128-generic kernel version PS2: sorry for the delay!

[FalcoXYZ]

@andreicupaciu on ETHonArm, correct?

[andreicupaciu]

@FalcoXYZ No, on x86_64 (AMD CPU).

[FalcoXYZ]

@diglos @dannywillems Got a Raspberry Pi 4 8GB model but I can't reproduce this error. Any more information from your end?

[mastilver]

Hi

I'm getting the same error. logs:

$ ./Nethermind.Runner     --config xdai     --JsonRpc.Enabled true     --HealthChecks.Enabled true     --HealthChecks.UIEnabled true     --JsonRpc.EnginePort=8551     --JsonRpc.JwtSecretFile=../jwtsecret/jwt.hex --Network.DiscoveryPort=31061 --Network.P2PPort=31061
2022-12-08 20-24-09.1763|Nethermind starting initialization.
2022-12-08 20-24-09.2126|Loading embedded plugins
2022-12-08 20-24-09.2127|  Found plugin type Nethermind.Consensus.AuRa.AuRaPlugin
2022-12-08 20-24-09.2127|  Found plugin type Nethermind.Consensus.Clique.CliquePlugin
2022-12-08 20-24-09.2128|  Found plugin type Nethermind.Consensus.Ethash.EthashPlugin
2022-12-08 20-24-09.2128|  Found plugin type Nethermind.Consensus.Ethash.NethDevPlugin
2022-12-08 20-24-09.2128|  Found plugin type Nethermind.Hive.HivePlugin
2022-12-08 20-24-09.2128|  Found plugin type Nethermind.UPnP.Plugin.UPnPPlugin
Resolved executing directory as /home/debian/gnosis/execution.
2022-12-08 20-24-09.2496|Loading 9 assemblies from /home/debian/gnosis/execution/plugins
2022-12-08 20-24-09.2498|Loading assembly Nethermind.HealthChecks
2022-12-08 20-24-09.2510|  Found plugin type Nethermind.HealthChecks
2022-12-08 20-24-09.2510|Loading assembly Nethermind.Mev
2022-12-08 20-24-09.2532|  Found plugin type Nethermind.Mev
2022-12-08 20-24-09.2532|Loading assembly Nethermind.Init
2022-12-08 20-24-09.2538|Loading assembly Nethermind.AccountAbstraction
2022-12-08 20-24-09.2555|  Found plugin type Nethermind.AccountAbstraction
2022-12-08 20-24-09.2555|Loading assembly Nethermind.Api
2022-12-08 20-24-09.2563|Loading assembly Nethermind.Merge.Plugin
2022-12-08 20-24-09.2580|  Found plugin type Nethermind.Merge.Plugin
2022-12-08 20-24-09.2581|Loading assembly Nethermind.EthStats
2022-12-08 20-24-09.2583|  Found plugin type Nethermind.EthStats
2022-12-08 20-24-09.2583|Loading assembly Nethermind.Merge.AuRa
2022-12-08 20-24-09.2586|  Found plugin type Nethermind.Merge.AuRa
2022-12-08 20-24-09.2586|Loading assembly Nethermind.Consensus.AuRa
2022-12-08 20-24-09.3140|Loading standard NLog.config file from /home/debian/gnosis/execution/NLog.config.
2022-12-08 20-24-09.4634|NLog.config loaded in 149ms.
2022-12-08 20-24-09.4648|Reading config file from /home/debian/gnosis/execution/configs/xdai.cfg
2022-12-08 20-24-09.6281|Configuration initialized.
2022-12-08 20:24:09.6501|RocksDb Version: 6.29.3 
2022-12-08 20:24:09.6612|Loading chainspec from embedded resources: /home/debian/gnosis/execution/chainspec/xdai.json 
2022-12-08 20:24:09.8985|Setting up memory allowances 
2022-12-08 20:24:09.8985|  memory hint:        768MB 
2022-12-08 20:24:09.8985|  general memory:     32MB 
2022-12-08 20:24:09.8985|  peers memory:       50MB 
2022-12-08 20:24:09.8985|  Netty memory:       100MB 
2022-12-08 20:24:09.8985|  mempool memory:     111MB 
2022-12-08 20:24:09.8985|  fast blocks memory: 47MB 
2022-12-08 20:24:09.9015|  trie memory:        85MB 
2022-12-08 20:24:09.9015|  DB memory:          341MB 
2022-12-08 20:24:09.9310|Using http://ipv4.icanhazip.com to get external ip 
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted

System: Linux version 4.19.0-18-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.208-1 (2021-09-29) Nethermind: nethermind-linux-amd64-1.14.7-4fe81c6

I don't have librocksdb-dev nor librocksdb5.17 installed. Let me know if you need more information, Thank you

[FalcoXYZ]

@mastilver can you share the exact system and OS you are using?

[mastilver]

@FalcoXYZ I'm running this server: https://www.ovhcloud.com/en-gb/bare-metal/rise/rise-1/ using a Intel Xeon-E 2136 - 6c/12t - 3.3 GHz/4.5 GHz running Debian 10 (Buster)

If I delete all the data and restart the install, the node is going to run for a while then crash, after that I get the same logs a above

[FalcoXYZ]

@mastilver does this only happen on Debian for you? If it isn't too much trouble, are you able to test with Ubuntu 22 and let me know if this also gives you the same error or not? (just want to make sure if this is OS issue or not)

[mastilver]

@FalcoXYZ I switched my server to Ubuntu 22 and it's now syncing without issues. I didn't test it on a fresh Debian, so it could still be because I had something incorrectly setup on my previous server

Thank you

[gituser]

Also got this error on the latest 1.17.0 on Ubuntu 18.04 LTS.

Nethermind was stuck in limbo with errors like:

2023-02-20 12:54:50.8527|INFO|1|RocksDb Version: 7.7.3 
2023-02-20 12:54:50.8619|WARN|1|ChainSpecPath matched an embedded resource inside the binary. Loading chainspec from embedded resources instead of file! 
2023-02-20 12:54:51.1031|ERROR|12|RocksDb was not found, please make sure it is installed on your machine. 
 On macOs : 'brew install rocksdb' System.TypeInitializationException: The type initializer for 'RocksDbSharp.Native' threw an exception.
 ---> NativeImport.NativeLoadException: Unable to locate rocksdb native library, either install it, or use RocksDbNative nuget package
Searched:
/home/nethermind/runtimes/linux-x64/native/librocksdb-7.7.3.so: (NativeLoadException) dlopen: /home/nethermind/runtimes/linux-x64/native/librocksdb-7.7.3.so: cannot open shared object file: No such file or directory
/home/nethermind/runtimes/linux-x64/native/librocksdb-7.7.so: (NativeLoadException) dlopen: /home/nethermind/runtimes/linux-x64/native/librocksdb-7.7.so: cannot open shared object file: No such file or directory
/home/nethermind/runtimes/linux-x64/native/librocksdb-7.so: (NativeLoadException) dlopen: /home/nethermind/runtimes/linux-x64/native/librocksdb-7.so: cannot open shared object file: No such file or directory
/home/nethermind/runtimes/linux-x64/native/librocksdb.so: (NativeLoadException) dlopen: /home/nethermind/runtimes/linux-x64/native/librocksdb.so: cannot open shared object file: No such file or directory
/home/nethermind/native/amd64/librocksdb-7.7.3.so: (NativeLoadException) dlopen: /home/nethermind/native/amd64/librocksdb-7.7.3.so: cannot open shared object file: No such file or directory
/home/nethermind/native/amd64/librocksdb-7.7.so: (NativeLoadException) dlopen: /home/nethermind/native/amd64/librocksdb-7.7.so: cannot open shared object file: No such file or directory
/home/nethermind/native/amd64/librocksdb-7.so: (NativeLoadException) dlopen: /home/nethermind/native/amd64/librocksdb-7.so: cannot open shared object file: No such file or directory
/home/nethermind/native/amd64/librocksdb.so: (NativeLoadException) dlopen: /home/nethermind/native/amd64/librocksdb.so: cannot open shared object file: No such file or directory
/home/nethermind/native/librocksdb-7.7.3.so: (NativeLoadException) dlopen: /home/nethermind/native/librocksdb-7.7.3.so: cannot open shared object file: No such file or directory
/home/nethermind/native/librocksdb-7.7.so: (NativeLoadException) dlopen: /home/nethermind/native/librocksdb-7.7.so: cannot open shared object file: No such file or directory
/home/nethermind/native/librocksdb-7.so: (NativeLoadException) dlopen: /home/nethermind/native/librocksdb-7.so: cannot open shared object file: No such file or directory
/home/nethermind/native/librocksdb.so: (NativeLoadException) dlopen: /home/nethermind/native/librocksdb.so: cannot open shared object file: No such file or directory
/home/nethermind/amd64/librocksdb-7.7.3.so: (NativeLoadException) dlopen: /home/nethermind/amd64/librocksdb-7.7.3.so: cannot open shared object file: No such file or directory
/home/nethermind/amd64/librocksdb-7.7.so: (NativeLoadException) dlopen: /home/nethermind/amd64/librocksdb-7.7.so: cannot open shared object file: No such file or directory
/home/nethermind/amd64/librocksdb-7.so: (NativeLoadException) dlopen: /home/nethermind/amd64/librocksdb-7.so: cannot open shared object file: No such file or directory
/home/nethermind/amd64/librocksdb.so: (NativeLoadException) dlopen: /home/nethermind/amd64/librocksdb.so: cannot open shared object file: No such file or directory
/home/nethermind/librocksdb-7.7.3.so: (NativeLoadException) dlopen: /home/nethermind/librocksdb-7.7.3.so: cannot open shared object file: No such file or directory
/home/nethermind/librocksdb-7.7.so: (NativeLoadException) dlopen: /home/nethermind/librocksdb-7.7.so: cannot open shared object file: No such file or directory

Removing / adding librocksdb and librocksdb-dev didn't help at all.

The only solution is:

• update to Ubuntu 20.04 (Focal)
• remove both librocksdb and librocksdb-dev packages

I'd suggest adding all dependencies including librocksdb bundled inside the package to avoid issues like these.

[MarekM25]

I think it was solved. We had many rocksdb upgrade. @gituser @flcl42@mastilver @andreicupaciu @whysthatso - please report new issue with reference to this one in case you still have this problem