cosmos-sdk
cosmos-sdk copied to clipboard
cosmos
Can't import/work with genesis.json larger 4GB
Hello, In GenesisL1 we have largest state, exported state is ~10GB. Snappy compression lib can't handle anything above ~4GB and throwing "panic: snappy: decoded block is too large" on importing genesis.json to new chain during upgrade. 4gb state might meet Cosmos and other chains, so maybe its a good idea to move to other compression lib. Thank you
Is the genesis.json entirely pre-compressed? Or only parts of the data inside genesis.json?
Maybe snappy is used during protobuf portion of JSON decode and not supporting > 4GB?
Is there a particular part of your genesis that is larger than others? Number of accounts? Contract state?
Can you post the backtrace dump of the panic?
Its a genesis.json obtained via "genesisd export" command which is a part of ethermint, so entire state. And this error comes when trying to add this genesis.json with cronos (and also kava) source codes. We have all parts of genesis.json large: more than 700k account (because of L1 coins airdrop/gift via solidity smart contract in our chain to Cosmos and Evmos delegators) and also very large contract state with about 1MLN NFT of molecular data (https://molnft.org) and and also few GB of molecular dataset written entirely onchain and indexed there with onchain search possibility: https://datasetnft.org/protcrud.
We expect also to increase that size to 20gb during recording of all obtained proteins (about 200k) entirely onchain (this will add about 10gb of data to store ~50 years of molecular biology experiments for public view/research).
Will post more info on panic in a few hours from that message, thank you!
This is with cronosd and imported genesis.json (same message on just exported ~10GB and with prettyfied ~14gb)
` cronosd start --log_level=trace 12:05PM INF Unlocking keyring 12:05PM INF starting ABCI with Tendermint 12:05PM INF starting node with ABCI Tendermint in-process panic: snappy: decoded block is too large
goroutine 1 [running]: github.com/golang/snappy.Encode({0xc001f96040?, 0x12?, 0x0?}, {0xca6da5c000?, 0x27661a0?, 0x43b8aa?}) github.com/golang/[email protected]/encode.go:22 +0x2d1 github.com/syndtr/goleveldb/leveldb/table.(*Writer).writeBlock(0xc001bf0000, 0xc001bf0060, 0xc001bf0060?) github.com/syndtr/[email protected]/leveldb/table/writer.go:172 +0x139 github.com/syndtr/goleveldb/leveldb/table.(*Writer).finishBlock(0xc001bf0000) github.com/syndtr/[email protected]/leveldb/table/writer.go:223 +0x46 github.com/syndtr/goleveldb/leveldb/table.(*Writer).Append(0xc001bf0000, {0xc803120000, 0x12, 0x26a93c000}, {0xc803120012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/[email protected]/leveldb/table/writer.go:256 +0x253 github.com/syndtr/goleveldb/leveldb.(*tWriter).append(0xc000da68a0, {0xc803120000, 0x12, 0x26a93c000}, {0xc803120012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/[email protected]/leveldb/table.go:558 +0x1e5 github.com/syndtr/goleveldb/leveldb.(*tOps).createFrom(0x26a93ad15?, {0x4be81b0, 0xc001194000}) github.com/syndtr/[email protected]/leveldb/table.go:397 +0x16d github.com/syndtr/goleveldb/leveldb.(*session).flushMemdb(0xc00160a000, 0xc000ddc280, 0xc0001b6a80, 0x0?) github.com/syndtr/[email protected]/leveldb/session_compaction.go:35 +0x116 github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal(0xc001db4700) github.com/syndtr/[email protected]/leveldb/db.go:597 +0xcb6 github.com/syndtr/goleveldb/leveldb.openDB(0xc00160a000) github.com/syndtr/[email protected]/leveldb/db.go:131 +0x43c github.com/syndtr/goleveldb/leveldb.Open({0x4be63d0?, 0xc0001fb810?}, 0x5ba700?) github.com/syndtr/[email protected]/leveldb/db.go:203 +0x188 github.com/syndtr/goleveldb/leveldb.OpenFile({0xc000500300?, 0x2c0b742?}, 0x0) github.com/syndtr/[email protected]/leveldb/db.go:225 +0x5d github.com/cometbft/cometbft-db.NewGoLevelDBWithOpts({0x2c0b742?, 0xc00062aee8?}, {0xc00203a030?, 0x0?}, 0xc000e12c30?) github.com/cometbft/[email protected]/goleveldb.go:32 +0x7c github.com/cometbft/cometbft-db.NewGoLevelDB(...) github.com/cometbft/[email protected]/goleveldb.go:27 github.com/cometbft/cometbft-db.init.0.func1({0x2c0b742?, 0xc000e3d5c0?}, {0xc00203a030?, 0x9?}) github.com/cometbft/[email protected]/goleveldb.go:15 +0x25 github.com/cometbft/cometbft-db.NewDB({0x2c0b742, 0x5}, {0xc000db2500, 0x9}, {0xc00203a030, 0x12}) github.com/cometbft/[email protected]/db.go:69 +0x11d github.com/tendermint/tendermint/node.DefaultDBProvider(0xc000512030) github.com/tendermint/[email protected]/node/node.go:78 +0xaf github.com/tendermint/tendermint/node.initDBs(0xc000237040, 0x46eefd8) github.com/tendermint/[email protected]/node/node.go:243 +0x129 github.com/tendermint/tendermint/node.NewNode(0xc000237040, {0x4bd0380, 0xc000ddc6e0}, 0xc002066970, {0x4bae640, 0xc0005122e8}, 0x1?, 0x646f558?, 0xc002066b80, {0x4bd58e0, ...}, ...) github.com/tendermint/[email protected]/node/node.go:718 +0x79 github.com/evmos/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x4bfd5c8, 0xc000db66f0}, 0x0, {0xc001ceae90, 0xc}, {0x4befe58, ...}, ...}, ...) github.com/evmos/[email protected]/server/start.go:378 +0xf87 github.com/evmos/ethermint/server.StartCmd.func2(0xc001c02000?, {0xc001ce6ec0?, 0x0?, 0x1?}) github.com/evmos/[email protected]/server/start.go:158 +0x225 github.com/spf13/cobra.(*Command).execute(0xc001c02000, {0xc001ce6ea0, 0x1, 0x1}) github.com/spf13/[email protected]/command.go:916 +0x862 github.com/spf13/cobra.(*Command).ExecuteC(0xc001bf2000) github.com/spf13/[email protected]/command.go:1044 +0x3bd github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/[email protected]/command.go:968 github.com/spf13/cobra.(*Command).ExecuteContext(...) github.com/spf13/[email protected]/command.go:961 github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x251a330?, {0x2c0bdd9, 0x6}, {0xc00061cc30, 0xd}) github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x20f main.main() github.com/crypto-org-chain/cronos/v2/cmd/cronosd/main.go:13 +0x38 `
This is with kava
` kava start --log_level=trace 12:08PM INF Unlocking keyring 12:08PM INF starting ABCI with Tendermint 12:08PM INF starting node with ABCI Tendermint in-process panic: snappy: decoded block is too large
goroutine 1 [running]: github.com/golang/snappy.Encode({0xc0018e4050?, 0x12?, 0x0?}, {0xca6d6bc000?, 0x26d0b80?, 0x41464a?}) github.com/golang/[email protected]/encode.go:22 +0x2d1 github.com/syndtr/goleveldb/leveldb/table.(*Writer).writeBlock(0xc001782480, 0xc0017824e0, 0xc0017824e0?) github.com/syndtr/[email protected]/leveldb/table/writer.go:172 +0x139 github.com/syndtr/goleveldb/leveldb/table.(*Writer).finishBlock(0xc001782480) github.com/syndtr/[email protected]/leveldb/table/writer.go:223 +0x46 github.com/syndtr/goleveldb/leveldb/table.(*Writer).Append(0xc001782480, {0xc802d80000, 0x12, 0x26a93c000}, {0xc802d80012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/[email protected]/leveldb/table/writer.go:256 +0x253 github.com/syndtr/goleveldb/leveldb.(*tWriter).append(0xc000626ae0, {0xc802d80000, 0x12, 0x26a93c000}, {0xc802d80012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/[email protected]/leveldb/table.go:558 +0x1e5 github.com/syndtr/goleveldb/leveldb.(*tOps).createFrom(0x26a93ad15?, {0x37f86b0, 0xc001530180}) github.com/syndtr/[email protected]/leveldb/table.go:397 +0x16d github.com/syndtr/goleveldb/leveldb.(*session).flushMemdb(0xc00003d3b0, 0xc00020a8c0, 0xc00018b420, 0x0?) github.com/syndtr/[email protected]/leveldb/session_compaction.go:35 +0x116 github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal(0xc000c43340) github.com/syndtr/[email protected]/leveldb/db.go:597 +0xcb6 github.com/syndtr/goleveldb/leveldb.openDB(0xc00003d3b0) github.com/syndtr/[email protected]/leveldb/db.go:131 +0x43c github.com/syndtr/goleveldb/leveldb.Open({0x37f6d90?, 0xc001351ea0?}, 0x5bc500?) github.com/syndtr/[email protected]/leveldb/db.go:203 +0x188 github.com/syndtr/goleveldb/leveldb.OpenFile({0xc001422c80?, 0x2b6bbd7?}, 0x0) github.com/syndtr/[email protected]/leveldb/db.go:225 +0x5d github.com/cometbft/cometbft-db.NewGoLevelDBWithOpts({0x2b6bbd7?, 0xc00063ef30?}, {0xc0018e5b90?, 0x0?}, 0xc0001fa8f0?) github.com/cometbft/[email protected]/goleveldb.go:32 +0x7c github.com/cometbft/cometbft-db.NewGoLevelDB(...) github.com/cometbft/[email protected]/goleveldb.go:27 github.com/cometbft/cometbft-db.init.0.func1({0x2b6bbd7?, 0xc000d2c3c0?}, {0xc0018e5b90?, 0x9?}) github.com/cometbft/[email protected]/goleveldb.go:15 +0x25 github.com/cometbft/cometbft-db.NewDB({0x2b6bbd7, 0x5}, {0xc00140e510, 0x9}, {0xc0018e5b90, 0x10}) github.com/cometbft/[email protected]/db.go:64 +0x11d github.com/tendermint/tendermint/node.DefaultDBProvider(0xc000fc1fc8) github.com/tendermint/[email protected]/node/node.go:78 +0xaf github.com/tendermint/tendermint/node.initDBs(0xc000162000, 0x32e6f50) github.com/tendermint/[email protected]/node/node.go:243 +0x129 github.com/tendermint/tendermint/node.NewNode(0xc000162000, {0x37e0150, 0xc00020a640}, 0xc001b6ba40, {0x37bc5e0, 0xc000fc1db8}, 0x1?, 0x5063a90?, 0xc001b6bcd0, {0x37e5690, ...}, ...) github.com/tendermint/[email protected]/node/node.go:718 +0x79 github.com/evmos/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x380dbe8, 0xc000221080}, 0x0, {0x0, 0x0}, {0x3800278, ...}, ...}, ...) github.com/evmos/[email protected]/server/start.go:373 +0xf87 github.com/evmos/ethermint/server.StartCmd.func2(0xc001a2e900?, {0xc001b6b140?, 0x0?, 0x1?}) github.com/evmos/[email protected]/server/start.go:157 +0x225 github.com/spf13/cobra.(*Command).execute(0xc001a2e900, {0xc001b6b120, 0x1, 0x1}) github.com/spf13/[email protected]/command.go:916 +0x862 github.com/spf13/cobra.(*Command).ExecuteC(0xc0019f8900) github.com/spf13/[email protected]/command.go:1044 +0x3bd github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/[email protected]/command.go:968 github.com/spf13/cobra.(*Command).ExecuteContext(...) github.com/spf13/[email protected]/command.go:961 github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x2575720?, {0x2b69a3d, 0x4}, {0xc0006189f0, 0xb}) github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x20f main.main() github.com/kava-labs/kava/cmd/kava/main.go:16 +0x38
`
genesis.json can be found here (prettified): 135.181.5.216/genesis.json
The issue is a bit further down the stack (not exactly cosmos-sdk)
It looks like snappy directly affects goleveldb storage. Goleveldb can turn off compression if using more disk space is acceptable. Since you already have a fork of cometbft-db this can be done without affect on other chains.
pebbledb will not work without breaking up batches (throws error "panic: pebble: batch too large: >= 4.0 G") - hard limit does NOT look adjustable.
Did not look into the other storage backends since the are experimental/less used.
Here's something to adjust in your code to get around the snappy problem and you can try again. The disk usage may not be significantly different without compression. (Try and share what you experience)
It appears there might be a problem parsing in the genesis.json contents (not a snappy problem). The genesis.json document is fully read by this point.
The data/state.db is about 9.7G at the time of panic (cronosd tag v1.0.9)
Make this change
https://github.com/cometbft/cometbft-db/blob/main/goleveldb.go#L26
func NewGoLevelDB(name string, dir string) (*GoLevelDB, error) {
return NewGoLevelDBWithOpts(name, dir, &opt.Options{Compression: opt.NoCompression})
}
The issue when using the genesis.json provided at 135.181.5.216/genesis.json is:
Unlocking keyring
starting ABCI with Tendermint
service start impl=multiAppConn module=proxy msg={} server=node
service start connection=query impl=localClient module=abci-client msg={} server=node
service start connection=snapshot impl=localClient module=abci-client msg={} server=node
service start connection=mempool impl=localClient module=abci-client msg={} server=node
service start connection=consensus impl=localClient module=abci-client msg={} server=node
service start impl=EventBus module=events msg={} server=node
service start impl=PubSub module=pubsub msg={} server=node
service start impl=IndexerService module=txindex msg={} server=node
ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 server=node software-version=1.0.9
ABCI Replay Blocks appHeight=0 module=consensus server=node stateHeight=0 storeHeight=0
initializing blockchain state from genesis.json
panic: value is nil
goroutine 80 [running]:
github.com/cosmos/cosmos-sdk/store/types.AssertValidValue(...)
github.com/cosmos/[email protected]/store/types/validity.go:13
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Set(0xc04364e660?, {0xc0db34a340?, 0x1?, 0x1?}, {0x0?, 0xed?, 0xed?})
github.com/cosmos/[email protected]/store/gaskv/store.go:49 +0x11c
github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.SetAccount({{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, {0x3c11fd0, ...}, ...}, ...}, ...)
github.com/cosmos/[email protected]/x/auth/keeper/account.go:82 +0x2c4
github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.InitGenesis({{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, {0x3c11fd0, ...}, ...}, ...}, ...)
github.com/cosmos/[email protected]/x/auth/keeper/genesis.go:23 +0x35e
github.com/cosmos/cosmos-sdk/x/auth.AppModule.InitGenesis({{}, {{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, ...}, ...}, ...}, ...)
github.com/cosmos/[email protected]/x/auth/module.go:144 +0x125
github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(_, {{0x3c2df78, 0xc00005c090}, {0x3c425a0, 0xc0001f0b00}, {{0x0, 0x0}, {0xc00150c040, 0xc}, 0x67049f, ...}, ...}, ...)
github.com/cosmos/[email protected]/types/module/module.go:322 +0x423
github.com/crypto-org-chain/cronos/app.(*App).InitChainer(_, {{0x3c2df78, 0xc00005c090}, {0x3c425a0, 0xc0001f0b00}, {{0x0, 0x0}, {0xc00150c040, 0xc}, 0x67049f, ...}, ...}, ...)
github.com/crypto-org-chain/cronos/app/app.go:835 +0x1cf
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc000c668c0, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...})
github.com/cosmos/[email protected]/baseapp/abci.go:71 +0x3f5
github.com/tendermint/tendermint/abci/client.(*localClient).InitChainSync(0xc000e22240, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...})
github.com/tendermint/[email protected]/abci/client/local_client.go:272 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).InitChainSync(0x0?, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...})
github.com/tendermint/[email protected]/proxy/app_conn.go:77 +0x55
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0x2a7450d, 0x7}}, {0xc00150c040, 0xc}, 0x67049f, 0x0, {{0x0, ...}, ...}, ...}, ...)
github.com/tendermint/[email protected]/consensus/replay.go:319 +0xd78
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc000de9990, {0x3c45640, 0xc000e300d0})
github.com/tendermint/[email protected]/consensus/replay.go:268 +0x3d4
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0x2a7450d, 0x7}}, {0xc00150c040, 0xc}, 0x67049f, 0x0, ...}, ...)
github.com/tendermint/[email protected]/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc00153c3c0, {0x3c29dc0, 0xc0015fb900}, 0xc00051e810, {0x3c08e20, 0xc000273158}, 0x1?, 0x53d2568?, 0xc00051ea30, {0x3c2f600, ...}, ...)
github.com/tendermint/[email protected]/node/node.go:779 +0x597
github.com/evmos/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x3c565e8, 0xc000e3db60}, 0x0, {0x0, 0x0}, {0x3c49d58, ...}, ...}, ...)
github.com/evmos/[email protected]/server/start.go:330 +0xb67
github.com/evmos/ethermint/server.StartCmd.func2.2()
github.com/evmos/[email protected]/server/start.go:144 +0x5e
github.com/evmos/ethermint/server.wrapCPUProfile.func2()
github.com/evmos/[email protected]/server/start.go:640 +0x29
created by github.com/evmos/ethermint/server.wrapCPUProfile
github.com/evmos/[email protected]/server/start.go:639 +0x37b
how to properly apply this? Can you please provide commit if you have time, thank you very much! "unc NewGoLevelDB(name string, dir string) (*GoLevelDB, error) { return NewGoLevelDBWithOpts(name, dir, &opt.Options{Compression: opt.NoCompression}) }
Since you suggested to turn off compression, did following:
versiondb/extsort/sort_test.go: SnappyCompression: true, versiondb/client/convert_to_sst.go: SnappyCompression: true, versiondb/client/restore_app_db.go: SnappyCompression: true,
Changed to false
Rebuild
Started with that genesis.json
Got similar error, but after 5m waiting and ~90GB RAM consumed (did on 300gb swap)
time cronosd start 9:49PM INF Unlocking keyring 9:49PM INF starting ABCI with Tendermint 9:49PM INF starting node with ABCI Tendermint in-process 9:54PM INF service start impl=multiAppConn module=proxy msg={} server=node 9:54PM INF service start connection=query impl=localClient module=abci-client msg={} server=node 9:54PM INF service start connection=snapshot impl=localClient module=abci-client msg={} server=node 9:54PM INF service start connection=mempool impl=localClient module=abci-client msg={} server=node 9:54PM INF service start connection=consensus impl=localClient module=abci-client msg={} server=node 9:54PM INF service start impl=EventBus module=events msg={} server=node 9:54PM INF service start impl=PubSub module=pubsub msg={} server=node 9:54PM INF service start impl=IndexerService module=txindex msg={} server=node 9:54PM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 server=node software-version=memiavl/v0.0.3-10-gc823329 9:54PM INF ABCI Replay Blocks appHeight=0 module=consensus server=node stateHeight=0 storeHeight=0 panic: snappy: decoded block is too large [recovered] panic: snappy: decoded block is too large [recovered] panic: snappy: decoded block is too large
goroutine 44 [running]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction.func1() github.com/syndtr/[email protected]/leveldb/db_compaction.go:763 +0xc5 panic({0x27f39a0, 0xc000136370}) runtime/panic.go:884 +0x213 github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact.func1() github.com/syndtr/[email protected]/leveldb/db_compaction.go:159 +0x9b panic({0x27f39a0, 0xc000136370}) runtime/panic.go:884 +0x213 github.com/golang/snappy.Encode({0xc001cb8048?, 0x12?, 0x0?}, {0xd2e1534000?, 0x27661a0?, 0x43b8aa?}) github.com/golang/[email protected]/encode.go:22 +0x2d1 github.com/syndtr/goleveldb/leveldb/table.(*Writer).writeBlock(0xc001678240, 0xc0016782a0, 0xc0016782a0?) github.com/syndtr/[email protected]/leveldb/table/writer.go:172 +0x139 github.com/syndtr/goleveldb/leveldb/table.(*Writer).finishBlock(0xc001678240) github.com/syndtr/[email protected]/leveldb/table/writer.go:223 +0x46 github.com/syndtr/goleveldb/leveldb/table.(*Writer).Append(0xc001678240, {0xd076bf8000, 0x12, 0x26a93c000}, {0xd076bf8012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/[email protected]/leveldb/table/writer.go:256 +0x253 github.com/syndtr/goleveldb/leveldb.(*tWriter).append(0xc002580660, {0xd076bf8000, 0x12, 0x26a93c000}, {0xd076bf8012, 0x26a93ad15, 0x26a93bfee}) github.com/syndtr/[email protected]/leveldb/table.go:558 +0x1e5 github.com/syndtr/goleveldb/leveldb.(*tOps).createFrom(0xc00120d860?, {0x4be81b0, 0xc000ee8800}) github.com/syndtr/[email protected]/leveldb/table.go:397 +0x16d github.com/syndtr/goleveldb/leveldb.(*session).flushMemdb(0xc000fbc3c0, 0xc0018ed220, 0xc000e68c40, 0xc000f84b10?) github.com/syndtr/[email protected]/leveldb/session_compaction.go:35 +0x116 github.com/syndtr/goleveldb/leveldb.(*DB).memCompaction.func1(0xc002580501?) github.com/syndtr/[email protected]/leveldb/db_compaction.go:305 +0xa5 github.com/syndtr/goleveldb/leveldb.(*compactionTransactFunc).run(0x0?, 0x0?) github.com/syndtr/[email protected]/leveldb/db_compaction.go:242 +0x1f github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact(0xc0016ca700, {0x2c16240, 0xb}, {0x4bb81b0, 0xc001cd0020}) github.com/syndtr/[email protected]/leveldb/db_compaction.go:186 +0x217 github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransactFunc(...) github.com/syndtr/[email protected]/leveldb/db_compaction.go:253 github.com/syndtr/goleveldb/leveldb.(*DB).memCompaction(0xc0016ca700) github.com/syndtr/[email protected]/leveldb/db_compaction.go:303 +0x3e8 github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0016ca700) github.com/syndtr/[email protected]/leveldb/db_compaction.go:777 +0x93 created by github.com/syndtr/goleveldb/leveldb.openDB github.com/syndtr/[email protected]/leveldb/db.go:156 +0x5d8
real 5m26.681s user 2m58.130s sys 1m19.140s
I have an open pr on cometbft that fixes this issue without needing to disable snappy compression: https://github.com/cometbft/cometbft/pull/1017
It removes saving the entire genesis json string into the database in a single key, which causes snappy to panic.
Disabling the snappy compression should work too with the cometbft-db modification above so that it avoids the step that causes it.
Amazing its already addressed by Kava, thank you! might we consider forking to Kava instead of Cronos. What do you think?
@drklee3 is correct. Not saving the entire genesis is another way to do this (but might break the /genesis API endpoint )
@alpha-omega-labs Sounds like you might work with Kava code, but if you need help making the suggested edit above to turn off compression, write back
@drklee3 is correct. Not saving the entire genesis is another way to do this (but might break the /genesis API endpoint )
@alpha-omega-labs Sounds like you might work with Kava code, but if you need help making the suggested edit above to turn off compression, write back
yes please, it would be very very helpful if you provide code way to disable compression in current cronosd or kava source code. Here is a direct code fork of cronos since we started to experiment with it because evmos went close source. https://github.com/alpha-omega-labs/genesisL1 If you can provide commit to disable compression there (or in kava) that would be highly appreciated! Thank you!
Here's something to adjust in your code to get around the snappy problem and you can try again. The disk usage may not be significantly different without compression. (Try and share what you experience)
It appears there might be a problem parsing in the genesis.json contents (not a snappy problem). The genesis.json document is fully read by this point.
The data/state.db is about 9.7G at the time of panic (cronosd tag v1.0.9)
Make this change
https://github.com/cometbft/cometbft-db/blob/main/goleveldb.go#L26
func NewGoLevelDB(name string, dir string) (*GoLevelDB, error) { return NewGoLevelDBWithOpts(name, dir, &opt.Options{Compression: opt.NoCompression}) }The issue when using the genesis.json provided at
135.181.5.216/genesis.jsonis:Unlocking keyring starting ABCI with Tendermint service start impl=multiAppConn module=proxy msg={} server=node service start connection=query impl=localClient module=abci-client msg={} server=node service start connection=snapshot impl=localClient module=abci-client msg={} server=node service start connection=mempool impl=localClient module=abci-client msg={} server=node service start connection=consensus impl=localClient module=abci-client msg={} server=node service start impl=EventBus module=events msg={} server=node service start impl=PubSub module=pubsub msg={} server=node service start impl=IndexerService module=txindex msg={} server=node ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 server=node software-version=1.0.9 ABCI Replay Blocks appHeight=0 module=consensus server=node stateHeight=0 storeHeight=0 initializing blockchain state from genesis.json panic: value is nil goroutine 80 [running]: github.com/cosmos/cosmos-sdk/store/types.AssertValidValue(...) github.com/cosmos/[email protected]/store/types/validity.go:13 github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Set(0xc04364e660?, {0xc0db34a340?, 0x1?, 0x1?}, {0x0?, 0xed?, 0xed?}) github.com/cosmos/[email protected]/store/gaskv/store.go:49 +0x11c github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.SetAccount({{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, {0x3c11fd0, ...}, ...}, ...}, ...) github.com/cosmos/[email protected]/x/auth/keeper/account.go:82 +0x2c4 github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.InitGenesis({{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, {0x3c11fd0, ...}, ...}, ...}, ...) github.com/cosmos/[email protected]/x/auth/keeper/genesis.go:23 +0x35e github.com/cosmos/cosmos-sdk/x/auth.AppModule.InitGenesis({{}, {{0x3c11f80, 0xc000dda4d0}, {0x3c41b90, 0xc000517d80}, {{0x3c41b90, 0xc000517d80}, 0xc00010f940, {0x3c11f80, 0xc000dda540}, ...}, ...}, ...}, ...) github.com/cosmos/[email protected]/x/auth/module.go:144 +0x125 github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(_, {{0x3c2df78, 0xc00005c090}, {0x3c425a0, 0xc0001f0b00}, {{0x0, 0x0}, {0xc00150c040, 0xc}, 0x67049f, ...}, ...}, ...) github.com/cosmos/[email protected]/types/module/module.go:322 +0x423 github.com/crypto-org-chain/cronos/app.(*App).InitChainer(_, {{0x3c2df78, 0xc00005c090}, {0x3c425a0, 0xc0001f0b00}, {{0x0, 0x0}, {0xc00150c040, 0xc}, 0x67049f, ...}, ...}, ...) github.com/crypto-org-chain/cronos/app/app.go:835 +0x1cf github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc000c668c0, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...}) github.com/cosmos/[email protected]/baseapp/abci.go:71 +0x3f5 github.com/tendermint/tendermint/abci/client.(*localClient).InitChainSync(0xc000e22240, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...}) github.com/tendermint/[email protected]/abci/client/local_client.go:272 +0x118 github.com/tendermint/tendermint/proxy.(*appConnConsensus).InitChainSync(0x0?, {{0x1, 0xed9ed0804, 0x0}, {0xc00150c040, 0xc}, 0xc000260180, {0xc000023200, 0x2c, 0x2c}, ...}) github.com/tendermint/[email protected]/proxy/app_conn.go:77 +0x55 github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0x2a7450d, 0x7}}, {0xc00150c040, 0xc}, 0x67049f, 0x0, {{0x0, ...}, ...}, ...}, ...) github.com/tendermint/[email protected]/consensus/replay.go:319 +0xd78 github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc000de9990, {0x3c45640, 0xc000e300d0}) github.com/tendermint/[email protected]/consensus/replay.go:268 +0x3d4 github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0x2a7450d, 0x7}}, {0xc00150c040, 0xc}, 0x67049f, 0x0, ...}, ...) github.com/tendermint/[email protected]/node/node.go:329 +0x1b8 github.com/tendermint/tendermint/node.NewNode(0xc00153c3c0, {0x3c29dc0, 0xc0015fb900}, 0xc00051e810, {0x3c08e20, 0xc000273158}, 0x1?, 0x53d2568?, 0xc00051ea30, {0x3c2f600, ...}, ...) github.com/tendermint/[email protected]/node/node.go:779 +0x597 github.com/evmos/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x3c565e8, 0xc000e3db60}, 0x0, {0x0, 0x0}, {0x3c49d58, ...}, ...}, ...) github.com/evmos/[email protected]/server/start.go:330 +0xb67 github.com/evmos/ethermint/server.StartCmd.func2.2() github.com/evmos/[email protected]/server/start.go:144 +0x5e github.com/evmos/ethermint/server.wrapCPUProfile.func2() github.com/evmos/[email protected]/server/start.go:640 +0x29 created by github.com/evmos/ethermint/server.wrapCPUProfile github.com/evmos/[email protected]/server/start.go:639 +0x37b
Yes, you are right! Was able to get your suggestion work on both kava and cronos and got that error as you do. genesis.json originally exported as single line and was prettified to be more human readable. Might this was an issue. Will try to put original genesis.json and see what happens.
Got same error with non modified at all genesis.json
At the same time genesis.json seems to be visibly good - where to dig for error?
initializing blockchain state from genesis.json panic: value is nil
goroutine 1 [running]: github.com/cosmos/cosmos-sdk/store/types.AssertValidValue(...) github.com/cosmos/[email protected]/store/types/validity.go:13 github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Set(0x61f2560?, {0xc0fd4560a0?, 0x1?, 0x1?}, {0x0?, 0xed?, 0xed?}) github.com/cosmos/[email protected]/store/gaskv/store.go:49 +0x11c github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.SetAccount({{0x4bb7548, 0xc000ee6c20}, {0x4be7b90, 0xc00139f980}, {{0x4be7b90, 0xc00139f980}, 0xc0000148b0, {0x4bb7548, 0xc000ee6c90}, {0x4bb7598, ...}, ...}, ...}, ...) github.com/cosmos/[email protected]/x/auth/keeper/account.go:82 +0x2da github.com/cosmos/cosmos-sdk/x/auth/keeper.AccountKeeper.InitGenesis({{0x4bb7548, 0xc000ee6c20}, {0x4be7b90, 0xc00139f980}, {{0x4be7b90, 0xc00139f980}, 0xc0000148b0, {0x4bb7548, 0xc000ee6c90}, {0x4bb7598, ...}, ...}, ...}, ...) github.com/cosmos/[email protected]/x/auth/keeper/genesis.go:23 +0x285 github.com/cosmos/cosmos-sdk/x/auth.AppModule.InitGenesis({{}, {{0x4bb7548, 0xc000ee6c20}, {0x4be7b90, 0xc00139f980}, {{0x4be7b90, 0xc00139f980}, 0xc0000148b0, {0x4bb7548, 0xc000ee6c90}, ...}, ...}, ...}, ...) github.com/cosmos/[email protected]/x/auth/module.go:144 +0x125 github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(, {{0x4bd4338, 0xc000056090}, {0x4be85a0, 0xc00183f6c0}, {{0x0, 0x0}, {0xc0019d2010, 0xc}, 0x67049f, ...}, ...}, ...) github.com/cosmos/[email protected]/types/module/module.go:322 +0x363 github.com/crypto-org-chain/cronos/v2/app.(*App).InitChainer(, {{0x4bd4338, 0xc000056090}, {0x4be85a0, 0xc00183f6c0}, {{0x0, 0x0}, {0xc0019d2010, 0xc}, 0x67049f, ...}, ...}, ...) github.com/crypto-org-chain/cronos/v2/app/app.go:836 +0x1cf github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc0015a8380, {{0x1, 0xed9ed0804, 0x0}, {0xc0019d2010, 0xc}, 0xc00013f420, {0xc000f0c000, 0x2c, 0x2c}, ...}) github.com/cosmos/[email protected]/baseapp/abci.go:71 +0x3f5 github.com/tendermint/tendermint/abci/client.(*localClient).InitChainSync(0xc00115ad20, {{0x1, 0xed9ed0804, 0x0}, {0xc0019d2010, 0xc}, 0xc00013f420, {0xc000f0c000, 0x2c, 0x2c}, ...}) github.com/tendermint/[email protected]/abci/client/local_client.go:272 +0x118 github.com/tendermint/tendermint/proxy.(*appConnConsensus).InitChainSync(0xc0001327b8?, {{0x1, 0xed9ed0804, 0x0}, {0xc0019d2010, 0xc}, 0xc00013f420, {0xc000f0c000, 0x2c, 0x2c}, ...}) github.com/tendermint/[email protected]/proxy/app_conn.go:77 +0x55 github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(, {{{0xb, 0x0}, {0x2c0d9fe, 0x7}}, {0xc0019d2010, 0xc}, 0x67049f, 0x0, {{0x0, ...}, ...}, ...}, ...) github.com/tendermint/[email protected]/consensus/replay.go:319 +0xf18 github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc00069cf68, {0x4beb4e8, 0xc0010ee0d0}) github.com/tendermint/[email protected]/consensus/replay.go:268 +0x3d4 github.com/tendermint/tendermint/node.doHandshake({, }, {{{0xb, 0x0}, {0x2c0d9fe, 0x7}}, {0xc0019d2010, 0xc}, 0x67049f, 0x0, ...}, ...) github.com/tendermint/[email protected]/node/node.go:329 +0x1b8 github.com/tendermint/tendermint/node.NewNode(0xc00014a8c0, {0x4bd0460, 0xc001505400}, 0xc000f26670, {0x4bae720, 0xc0010f2df8}, 0x1?, 0x646f558?, 0xc000f26880, {0x4bd59c0, ...}, ...) github.com/tendermint/[email protected]/node/node.go:779 +0x597 github.com/evmos/ethermint/server.startInProcess(, {{0x0, 0x0, 0x0}, {0x4bfd6a8, 0xc0012c05d0}, 0x0, {0x0, 0x0}, {0x4beff38, ...}, ...}, ...) github.com/evmos/[email protected]/server/start.go:378 +0xf87 github.com/evmos/ethermint/server.StartCmd.func2(0xc000f22000?, {0x646f558?, 0x0?, 0x0?}) github.com/evmos/[email protected]/server/start.go:158 +0x225 github.com/spf13/cobra.(*Command).execute(0xc000f22000, {0x646f558, 0x0, 0x0}) github.com/spf13/[email protected]/command.go:916 +0x862 github.com/spf13/cobra.(*Command).ExecuteC(0xc000f14000) github.com/spf13/[email protected]/command.go:1044 +0x3bd github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/[email protected]/command.go:968 github.com/spf13/cobra.(*Command).ExecuteContext(...) github.com/spf13/[email protected]/command.go:961 github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x251a350?, {0x2c0bdd7, 0x6}, {0xc001171520, 0xd}) github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x20f main.main() github.com/crypto-org-chain/cronos/v2/cmd/cronosd/main.go:13 +0x38
searching for a null in genesis.json returned: Null value at app_state.authz.authorization.1.authorization.max_tokens ... ... ... Null value at app_state.authz.authorization.1355.authorization.max_tokens
Null value at app_state.ibc.client_genesis.clients.0.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.0.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.1.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.1.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.2.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.2.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.3.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.3.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.4.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.4.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.5.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.5.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.6.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.6.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.7.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.7.client_state.proof_specs.1.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.8.client_state.proof_specs.0.inner_spec.empty_child Null value at app_state.ibc.client_genesis.clients.8.client_state.proof_specs.1.inner_spec.empty_child
Null value at app_state.params
Where is most likely fail do you think?