osmosis [Bug]: "panic: unexpected EOF" upon startup

Running on Ubuntu 22.04.3 LTS

What happened?

Starting cosmovisor through systemctl fails with this log output -

Dec 21 10:28:10 pc systemd[1]: Started Cosmovisor daemon.
Dec 21 10:28:10 pc cosmovisor[239947]: 10:28AM INF running app args=["start","--home","/blockchains/osmosis"] module=cosmovisor path=/blockchains/osmosis/cosmovisor/upgrades/v21/bin/osmosisd
Dec 21 10:28:10 pc cosmovisor[239975]: 10:28AM INF starting ABCI with Tendermint
Dec 21 10:28:11 pc cosmovisor[239975]: 10:28AM INF service start impl=multiAppConn module=proxy msg={}
Dec 21 10:28:11 pc cosmovisor[239975]: 10:28AM INF service start connection=query impl=localClient module=abci-client msg={}
Dec 21 10:28:11 pc cosmovisor[239975]: 10:28AM INF service start connection=snapshot impl=localClient module=abci-client msg={}
Dec 21 10:28:11 pc cosmovisor[239975]: 10:28AM INF service start connection=mempool impl=localClient module=abci-client msg={}
Dec 21 10:28:11 pc cosmovisor[239975]: 10:28AM INF service start connection=consensus impl=localClient module=abci-client msg={}
Dec 21 10:28:11 pc cosmovisor[239975]: 10:28AM INF service start impl=EventBus module=events msg={}
Dec 21 10:28:11 pc cosmovisor[239975]: 10:28AM INF service start impl=PubSub module=pubsub msg={}
Dec 21 10:28:12 pc cosmovisor[239975]: 10:28AM INF service start impl=IndexerService module=txindex msg={}
Dec 21 10:28:12 pc cosmovisor[239975]: 10:28AM INF ABCI Handshake App Info hash="Q!筍c���\x11�����]���׳NN��E����ck" height=12872745 module=consensus protocol-version=20 software-version=20.2.1
Dec 21 10:28:12 pc cosmovisor[239975]: 10:28AM INF ABCI Replay Blocks appHeight=12872745 module=consensus stateHeight=12872745 storeHeight=12872746
Dec 21 10:28:12 pc cosmovisor[239975]: 10:28AM INF Replay last block using real app module=consensus
Dec 21 10:28:12 pc cosmovisor[239975]: panic: unexpected EOF
Dec 21 10:28:12 pc cosmovisor[239975]: goroutine 1 [running]:
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/codec.(*ProtoCodec).MustUnmarshal(0x0?, {0xc0088850f0?, 0x0?, 0x336a4d8?}, {0x3364418?, 0xc00889b188?})
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/codec/proto_codec.go:97 +0x45
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/x/bank/keeper.BaseViewKeeper.IterateAccountBalances({{_, _}, {_, _}, {_, _}}, {{0x3358488, 0xc000056058}, {0x336de60, 0xc0056f3ec0}, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/x/bank/keeper/view.go:124 +0x27b
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/x/bank/keeper.BaseViewKeeper.GetAllBalances({{_, _}, {_, _}, {_, _}}, {{0x3358488, 0xc000056058}, {0x336de60, 0xc0056f3ec0}, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/x/bank/keeper/view.go:63 +0x146
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokens({{0x3338268, 0xc0015f7b10}, {0x336bbc0, 0xc0004d2e80}, {{0x336bbc0, 0xc0004d2e80}, 0xc0013c6ae0, {0x3338268, 0xc0015f7b50}, {0x33382b8, ...}, ...}, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/x/distribution/keeper/allocation.go:27 +0x1c2
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/x/distribution.BeginBlocker({{0x3358488, 0xc000056058}, {0x336de60, 0xc0056f3ec0}, {{0xb, 0x14}, {0xc003089ce0, 0x9}, 0xc46c2a, {0x24195648, ...}, ...}, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/x/distribution/abci.go:32 +0x214
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/x/distribution.AppModule.BeginBlock(...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/x/distribution/module.go:170
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x3358488, 0xc000056058}, {0x336de60, 0xc0056f3ec0}, {{0xb, 0x14}, {0xc003089ce0, 0x9}, 0xc46c2a, ...}, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/types/module/module.go:500 +0x178
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/osmosis-labs/osmosis/v20/app.(*OsmosisApp).BeginBlocker(_, {{0x3358488, 0xc000056058}, {0x336de60, 0xc0056f3ec0}, {{0xb, 0x14}, {0xc003089ce0, 0x9}, 0xc46c2a, ...}, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/osmosis-labs/osmosis/v20/app/app.go:362 +0xbb
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc004e7f280, 0x20, 0x20}, {{0xb, 0x14}, {0xc003089ce0, 0x9}, 0xc46c2a, {0x24195648, ...}, ...}, ...})
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/baseapp/abci.go:226 +0x8c3
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc004e7f280, 0x20, 0x20}, {{0xb, 0x14}, {0xc003089ce0, 0x9}, 0xc46c2a, {0x24195648, ...}, ...}, ...})
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/tendermint/[email protected]/abci/client/local_client.go:280 +0x118
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc004e7f280, 0x20, 0x20}, {{0xb, 0x14}, {0xc003089ce0, 0x9}, 0xc46c2a, {0x24195648, ...}, ...}, ...})
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/tendermint/[email protected]/proxy/app_conn.go:81 +0x55
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x3359ad8?, 0xc005f6b920}, {0x3362ed0, 0xc000fc90e0}, 0xc0001a83c0, {0x336d2e8, 0xc005efe078}, 0xc46c29?)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/tendermint/[email protected]/state/execution.go:307 +0x51d
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x14}, {0xc003088330, 0x7}}, {0xc003088337, 0x9}, 0x1, 0xc46c29, {{0xc00010f580, ...}, ...}, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/tendermint/[email protected]/state/execution.go:140 +0x171
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x14}, {0xc003088330, 0x7}}, {0xc003088337, 0x9}, 0x1, 0xc46c29, {{0xc00010f580, ...}, ...}, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/tendermint/[email protected]/consensus/replay.go:503 +0x23c
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x14}, {0xc003088330, 0x7}}, {0xc003088337, 0x9}, 0x1, 0xc46c29, {{0xc00010f580, ...}, ...}, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/tendermint/[email protected]/consensus/replay.go:416 +0x7ae
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc00272f760, {0x336e7c8, 0xc000fb6340})
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/tendermint/[email protected]/consensus/replay.go:268 +0x3d4
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x14}, {0xc003088330, 0x7}}, {0xc003088337, 0x9}, 0x1, 0xc46c29, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/tendermint/[email protected]/node/node.go:329 +0x1b8
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/tendermint/tendermint/node.NewNode(0xc0012c6000, {0x33533d0, 0xc000ea4960}, 0xc00166a480, {0x3330740, 0xc005efef90}, 0x0?, 0x1?, 0xc00166a620, {0x3359ad8, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/tendermint/[email protected]/node/node.go:777 +0x597
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x337d3f0, 0xc001294ae0}, {0x0, 0x0}, {0x335f6f8, 0xc0011748e0}, ...}, ...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/server/start.go:266 +0x85b
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc000e32000?, {0xc0010f6d80?, 0x0?, 0x2?})
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/server/start.go:129 +0x193
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/spf13/cobra.(*Command).execute(0xc000e32000, {0xc0010f6d60, 0x2, 0x2})
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/spf13/[email protected]/command.go:940 +0x862
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000806000)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/spf13/[email protected]/command.go:1068 +0x3bd
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/spf13/cobra.(*Command).Execute(...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/spf13/[email protected]/command.go:992
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/spf13/[email protected]/command.go:985
Dec 21 10:28:12 pc cosmovisor[239975]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc00126dab8, 0x13})
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x1eb
Dec 21 10:28:12 pc cosmovisor[239975]: main.main()
Dec 21 10:28:12 pc cosmovisor[239975]:         github.com/osmosis-labs/osmosis/v20/cmd/osmosisd/main.go:16 +0x31
Dec 21 10:28:12 pc cosmovisor[239947]: Error: exit status 2
Dec 21 10:28:12 pc cosmovisor[239947]: 10:28AM ERR error="exit status 2" module=cosmovisor
Dec 21 10:28:12 pc systemd[1]: cosmovisor.service: Main process exited, code=exited, status=1/FAILURE
Dec 21 10:28:12 pc systemd[1]: cosmovisor.service: Failed with result 'exit-code'.
Dec 21 10:28:12 pc systemd[1]: cosmovisor.service: Consumed 1.868s CPU time.

Starting osmosisd start produces the same error -

10:32AM INF starting ABCI with Tendermint
10:32AM INF service start impl=multiAppConn module=proxy msg={}
10:32AM INF service start connection=query impl=localClient module=abci-client msg={}
10:32AM INF service start connection=snapshot impl=localClient module=abci-client msg={}
10:32AM INF service start connection=mempool impl=localClient module=abci-client msg={}
10:32AM INF service start connection=consensus impl=localClient module=abci-client msg={}
10:32AM INF service start impl=EventBus module=events msg={}
10:32AM INF service start impl=PubSub module=pubsub msg={}
10:32AM INF service start impl=IndexerService module=txindex msg={}
10:32AM INF ABCI Handshake App Info hash="Q!筍c���\x11�����]���׳NN��E����ck" height=12872745 module=consensus protocol-version=20 software-version=20.2.1
10:32AM INF ABCI Replay Blocks appHeight=12872745 module=consensus stateHeight=12872745 storeHeight=12872746
10:32AM INF Replay last block using real app module=consensus
panic: unexpected EOF

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/codec.(*ProtoCodec).MustUnmarshal(0x0?, {0xc0085e2160?, 0x0?, 0x336a4d8?}, {0x3364418?, 0xc0085f4558?})
	github.com/cosmos/[email protected]/codec/proto_codec.go:97 +0x45
github.com/cosmos/cosmos-sdk/x/bank/keeper.BaseViewKeeper.IterateAccountBalances({{_, _}, {_, _}, {_, _}}, {{0x3358488, 0xc0000560b0}, {0x336de60, 0xc005637d00}, ...}, ...)
	github.com/cosmos/[email protected]/x/bank/keeper/view.go:124 +0x27b
github.com/cosmos/cosmos-sdk/x/bank/keeper.BaseViewKeeper.GetAllBalances({{_, _}, {_, _}, {_, _}}, {{0x3358488, 0xc0000560b0}, {0x336de60, 0xc005637d00}, ...}, ...)
	github.com/cosmos/[email protected]/x/bank/keeper/view.go:63 +0x146
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokens({{0x3338268, 0xc0016cdad0}, {0x336bbc0, 0xc0015ba490}, {{0x336bbc0, 0xc0015ba490}, 0xc001482dc8, {0x3338268, 0xc0016cdb10}, {0x33382b8, ...}, ...}, ...}, ...)
	github.com/cosmos/[email protected]/x/distribution/keeper/allocation.go:27 +0x1c2
github.com/cosmos/cosmos-sdk/x/distribution.BeginBlocker({{0x3358488, 0xc0000560b0}, {0x336de60, 0xc005637d00}, {{0xb, 0x14}, {0xc0066b6b90, 0x9}, 0xc46c2a, {0x24195648, ...}, ...}, ...}, ...)
	github.com/cosmos/[email protected]/x/distribution/abci.go:32 +0x214
github.com/cosmos/cosmos-sdk/x/distribution.AppModule.BeginBlock(...)
	github.com/cosmos/[email protected]/x/distribution/module.go:170
github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x3358488, 0xc0000560b0}, {0x336de60, 0xc005637d00}, {{0xb, 0x14}, {0xc0066b6b90, 0x9}, 0xc46c2a, ...}, ...}, ...)
	github.com/cosmos/[email protected]/types/module/module.go:500 +0x178
github.com/osmosis-labs/osmosis/v20/app.(*OsmosisApp).BeginBlocker(_, {{0x3358488, 0xc0000560b0}, {0x336de60, 0xc005637d00}, {{0xb, 0x14}, {0xc0066b6b90, 0x9}, 0xc46c2a, ...}, ...}, ...)
	github.com/osmosis-labs/osmosis/v20/app/app.go:362 +0xbb
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc002a49b20, 0x20, 0x20}, {{0xb, 0x14}, {0xc0066b6b90, 0x9}, 0xc46c2a, {0x24195648, ...}, ...}, ...})
	github.com/cosmos/[email protected]/baseapp/abci.go:226 +0x8c3
github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc002a49b20, 0x20, 0x20}, {{0xb, 0x14}, {0xc0066b6b90, 0x9}, 0xc46c2a, {0x24195648, ...}, ...}, ...})
	github.com/tendermint/[email protected]/abci/client/local_client.go:280 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc002a49b20, 0x20, 0x20}, {{0xb, 0x14}, {0xc0066b6b90, 0x9}, 0xc46c2a, {0x24195648, ...}, ...}, ...})
	github.com/tendermint/[email protected]/proxy/app_conn.go:81 +0x55
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x3359ad8?, 0xc0032be780}, {0x3362ed0, 0xc0011aade0}, 0xc00122a3c0, {0x336d2e8, 0xc0061f74e8}, 0xc46c29?)
	github.com/tendermint/[email protected]/state/execution.go:307 +0x51d
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x14}, {0xc00623c218, 0x7}}, {0xc00623c260, 0x9}, 0x1, 0xc46c29, {{0xc00620c040, ...}, ...}, ...}, ...)
	github.com/tendermint/[email protected]/state/execution.go:140 +0x171
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x14}, {0xc00623c218, 0x7}}, {0xc00623c260, 0x9}, 0x1, 0xc46c29, {{0xc00620c040, ...}, ...}, ...}, ...)
	github.com/tendermint/[email protected]/consensus/replay.go:503 +0x23c
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x14}, {0xc00623c218, 0x7}}, {0xc00623c260, 0x9}, 0x1, 0xc46c29, {{0xc00620c040, ...}, ...}, ...}, ...)
	github.com/tendermint/[email protected]/consensus/replay.go:416 +0x7ae
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc002a6b760, {0x336e7c8, 0xc000594dd0})
	github.com/tendermint/[email protected]/consensus/replay.go:268 +0x3d4
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x14}, {0xc00623c218, 0x7}}, {0xc00623c260, 0x9}, 0x1, 0xc46c29, ...}, ...)
	github.com/tendermint/[email protected]/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc0017328c0, {0x33533d0, 0xc000220280}, 0xc001611650, {0x3330740, 0xc0061f72a8}, 0x0?, 0x1?, 0xc0016117d0, {0x3359ad8, ...}, ...)
	github.com/tendermint/[email protected]/node/node.go:777 +0x597
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x337d3f0, 0xc00172e9c0}, {0x0, 0x0}, {0x335f6f8, 0xc001610d00}, ...}, ...)
	github.com/cosmos/[email protected]/server/start.go:266 +0x85b
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc0016bc600?, {0xc00121e180?, 0x0?, 0x2?})
	github.com/cosmos/[email protected]/server/start.go:129 +0x193
github.com/spf13/cobra.(*Command).execute(0xc0016bc600, {0xc00121e160, 0x2, 0x2})
	github.com/spf13/[email protected]/command.go:940 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc00163c600)
	github.com/spf13/[email protected]/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	github.com/spf13/[email protected]/command.go:985
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc0014b4f78, 0x13})
	github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x1eb
main.main()
	github.com/osmosis-labs/osmosis/v20/cmd/osmosisd/main.go:16 +0x31

Osmosis Version

20.2.1

How to reproduce?

Following instructions at https://docs.osmosis.zone/overview/validate/joining-mainnet/ -

curl -sL https://get.osmosis.zone/install > i.py && python3 i.py
During install - osmosis-1 network, install node, default purning, cosmovisor as a service. Reproduced either on Germany pruned or Netherland pruned.
Edit /usr/lib/systemd/system/cosmovisor.service and fix ExecStart=/usr/local/bin/cosmovisor start to ExecStart=/usr/local/bin/cosmovisor run start
sudo systemctl daemon-reload ; sudo systemctl restart cosmovisor

then the log output mentioned above appears.

Dec 21 '23 10:12 ko0f

Also tried to -

Delete the data folder ~/.osmosis
osmosisd init pc
wget -O ~/.osmosisd/config/genesis.json https://github.com/osmosis-labs/networks/raw/main/osmosis-1/genesis.json
osmosisd start

and it produces an error -

panic: runtime error: invalid memory address or nil pointer dereference

full log -

1:02PM INF starting ABCI with Tendermint
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196280, mint}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001964a0, interchainquery}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001962b0, slashing}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196450, wasm}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001964c0, cosmwasmpool}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196490, hooks-for-ibc}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001964b0, packetfowardmiddleware}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196350, twap}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196480, protorev}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196320, transfer}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196370, incentives}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001962f0, icahost}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196310, evidence}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196290, distribution}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196330, capability}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196390, poolincentives}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196460, tokenfactory}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196260, bank}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196270, staking}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196360, lockup}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196430, txfees}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001963a0, concentratedliquidity}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196440, superfluid}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001962d0, params}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001962e0, ibc}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196230, acc}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196340, gamm}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196300, upgrade}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001963b0, poolmanager}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001963c0, authz}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196380, epochs}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001962a0, downtimedetector}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc0001962c0, gov}" version=0
1:02PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D is_lazy=false store_key="KVStoreKey{0xc000196470, valsetpref}" version=0
1:02PM INF service start impl=multiAppConn module=proxy msg={}
1:02PM INF service start connection=query impl=localClient module=abci-client msg={}
1:02PM INF service start connection=snapshot impl=localClient module=abci-client msg={}
1:02PM INF service start connection=mempool impl=localClient module=abci-client msg={}
1:02PM INF service start connection=consensus impl=localClient module=abci-client msg={}
1:02PM INF service start impl=EventBus module=events msg={}
1:02PM INF service start impl=PubSub module=pubsub msg={}
1:02PM INF service start impl=IndexerService module=txindex msg={}
1:02PM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 software-version=20.2.1
1:02PM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
1:02PM INF initializing blockchain state from genesis.json
1:02PM INF running initialization for module module=capability
1:02PM INF running initialization for module module=auth
1:02PM INF running initialization for module module=bank
1:02PM INF running initialization for module module=distribution
1:02PM INF running initialization for module module=staking
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x12776dd]

goroutine 1 [running]:
math/big.(*Int).Sign(...)
	math/big/int.go:41
cosmossdk.io/math.Int.IsNegative(...)
	cosmossdk.io/[email protected]/int.go:206
github.com/cosmos/cosmos-sdk/x/staking/types.validateMinSelfDelegation({0x2b916e0?, 0x0?})
	github.com/cosmos/[email protected]/x/staking/types/params.go:254 +0x3d
github.com/cosmos/cosmos-sdk/x/params/types.Subspace.SetParamSet({{0x336bbc0, 0xc000165b00}, 0xc000014dd0, {0x3338268, 0xc0001962d0}, {0x33382b8, 0xc0001964d0}, {0xc0016de3a8, 0x7, 0x18}, ...}, ...)
	github.com/cosmos/[email protected]/x/params/types/subspace.go:236 +0x20d
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.SetParams(...)
	github.com/cosmos/[email protected]/x/staking/keeper/params.go:77
github.com/cosmos/cosmos-sdk/x/staking.InitGenesis({{0x3358488, 0xc0000560b0}, {0x336de60, 0xc00145b6c0}, {{0x0, 0x0}, {0xc0004a6170, 0x9}, 0x0, {0x0, ...}, ...}, ...}, ...)
	github.com/cosmos/[email protected]/x/staking/genesis.go:35 +0x1ec
github.com/cosmos/cosmos-sdk/x/staking.AppModule.InitGenesis({{{0x3371c80, 0xc000165b00}}, {{0x3338268, 0xc000196270}, {0x336bbc0, 0xc000165b00}, {0x335b1a0, 0xc0011c82d0}, {0x33688a0, 0xc001770000}, ...}, ...}, ...)
	github.com/cosmos/[email protected]/x/staking/module.go:154 +0x145
github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(_, {{0x3358488, 0xc0000560b0}, {0x336de60, 0xc00145b6c0}, {{0x0, 0x0}, {0xc0004a6170, 0x9}, 0x0, ...}, ...}, ...)
	github.com/cosmos/[email protected]/types/module/module.go:322 +0x35d
github.com/osmosis-labs/osmosis/v20/app.(*OsmosisApp).InitChainer(_, {{0x3358488, 0xc0000560b0}, {0x336de60, 0xc00145b6c0}, {{0x0, 0x0}, {0xc0004a6170, 0x9}, 0x0, ...}, ...}, ...)
	github.com/osmosis-labs/osmosis/v20/app/app.go:379 +0x1b5
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc000005b00, {{0x0, 0xed85ec810, 0x0}, {0xc0004a6170, 0x9}, 0xc001594cc0, {0x49842a0, 0x0, 0x0}, ...})
	github.com/cosmos/[email protected]/baseapp/abci.go:87 +0x515
github.com/tendermint/tendermint/abci/client.(*localClient).InitChainSync(0xc001b80ba0, {{0x0, 0xed85ec810, 0x0}, {0xc0004a6170, 0x9}, 0xc001594cc0, {0x49842a0, 0x0, 0x0}, ...})
	github.com/tendermint/[email protected]/abci/client/local_client.go:272 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).InitChainSync(0xc001462250?, {{0x0, 0xed85ec810, 0x0}, {0xc0004a6170, 0x9}, 0xc001594cc0, {0x49842a0, 0x0, 0x0}, ...})
	github.com/tendermint/[email protected]/proxy/app_conn.go:77 +0x55
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0x2bf1a25, 0x7}}, {0xc0004a6170, 0x9}, 0x1, 0x0, {{0x0, ...}, ...}, ...}, ...)
	github.com/tendermint/[email protected]/consensus/replay.go:319 +0xf18
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc0013a7760, {0x336e7c8, 0xc001206270})
	github.com/tendermint/[email protected]/consensus/replay.go:268 +0x3d4
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0x2bf1a25, 0x7}}, {0xc0004a6170, 0x9}, 0x1, 0x0, ...}, ...)
	github.com/tendermint/[email protected]/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc00128e000, {0x33533d0, 0xc001ba8c80}, 0xc00124f220, {0x3330740, 0xc0004b7278}, 0x0?, 0x1?, 0xc00124f410, {0x3359ad8, ...}, ...)
	github.com/tendermint/[email protected]/node/node.go:777 +0x597
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x337d3f0, 0xc00127eab0}, {0x0, 0x0}, {0x335f6f8, 0xc0011a2750}, ...}, ...)
	github.com/cosmos/[email protected]/server/start.go:266 +0x85b
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc000da8000?, {0x49842a0?, 0x0?, 0x0?})
	github.com/cosmos/[email protected]/server/start.go:129 +0x193
github.com/spf13/cobra.(*Command).execute(0xc000da8000, {0x49842a0, 0x0, 0x0})
	github.com/spf13/[email protected]/command.go:940 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc000d86000)
	github.com/spf13/[email protected]/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	github.com/spf13/[email protected]/command.go:985
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc0001f1590, 0x13})
	github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x1eb
main.main()
	github.com/osmosis-labs/osmosis/v20/cmd/osmosisd/main.go:16 +0x31

Dec 21 '23 13:12 ko0f

I've tried this with a non pruned snapshot (default), got the same error, a bit different stack trace -

panic: unexpected EOF

goroutine 81 [running]:
github.com/cosmos/cosmos-sdk/codec.(*ProtoCodec).MustUnmarshal(0x0?, {0xc14c381190?, 0x0?, 0x336a4d8?}, {0x3364418?, 0xc14c384e58?})
        github.com/cosmos/[email protected]/codec/proto_codec.go:97 +0x45
github.com/cosmos/cosmos-sdk/x/bank/keeper.BaseViewKeeper.IterateAccountBalances({{_, _}, {_, _}, {_, _}}, {{0x3358488, 0xc0000560f0}, {0x336de60, 0xc1490b9bc0}, ...}, ...)
        github.com/cosmos/[email protected]/x/bank/keeper/view.go:124 +0x27b
github.com/cosmos/cosmos-sdk/x/bank/keeper.BaseViewKeeper.GetAllBalances({{_, _}, {_, _}, {_, _}}, {{0x3358488, 0xc0000560f0}, {0x336de60, 0xc1490b9bc0}, ...}, ...)
        github.com/cosmos/[email protected]/x/bank/keeper/view.go:63 +0x146
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.AllocateTokens({{0x3338268, 0xc0055d9e00}, {0x336bbc0, 0xc0004dae80}, {{0x336bbc0, 0xc0004dae80}, 0xc00013c7e8, {0x3338268, 0xc0055d9e40}, {0x33382b8, ...}, ...}, ...}, ...)
        github.com/cosmos/[email protected]/x/distribution/keeper/allocation.go:27 +0x1c2
github.com/cosmos/cosmos-sdk/x/distribution.BeginBlocker({{0x3358488, 0xc0000560f0}, {0x336de60, 0xc1490b9bc0}, {{0xb, 0x14}, {0xc13f5f1960, 0x9}, 0xc469c5, {0x6731378, ...}, ...}, ...}, ...)
        github.com/cosmos/[email protected]/x/distribution/abci.go:32 +0x214
github.com/cosmos/cosmos-sdk/x/distribution.AppModule.BeginBlock(...)
        github.com/cosmos/[email protected]/x/distribution/module.go:170
github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x3358488, 0xc0000560f0}, {0x336de60, 0xc1490b9bc0}, {{0xb, 0x14}, {0xc13f5f1960, 0x9}, 0xc469c5, ...}, ...}, ...)
        github.com/cosmos/[email protected]/types/module/module.go:500 +0x178
github.com/osmosis-labs/osmosis/v20/app.(*OsmosisApp).BeginBlocker(_, {{0x3358488, 0xc0000560f0}, {0x336de60, 0xc1490b9bc0}, {{0xb, 0x14}, {0xc13f5f1960, 0x9}, 0xc469c5, ...}, ...}, ...)
        github.com/osmosis-labs/osmosis/v20/app/app.go:362 +0xbb
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc14905eba0, 0x20, 0x20}, {{0xb, 0x14}, {0xc13f5f1960, 0x9}, 0xc469c5, {0x6731378, ...}, ...}, ...})
        github.com/cosmos/[email protected]/baseapp/abci.go:226 +0x8c3
github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc14905eba0, 0x20, 0x20}, {{0xb, 0x14}, {0xc13f5f1960, 0x9}, 0xc469c5, {0x6731378, ...}, ...}, ...})
        github.com/tendermint/[email protected]/abci/client/local_client.go:280 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc14905eba0, 0x20, 0x20}, {{0xb, 0x14}, {0xc13f5f1960, 0x9}, 0xc469c5, {0x6731378, ...}, ...}, ...})
        github.com/tendermint/[email protected]/proxy/app_conn.go:81 +0x55
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x3359ad8?, 0xc000fb2d20}, {0x3362ed0, 0xc00136a3d0}, 0xc1269663c0, {0x336d2e8, 0xc00fdb1038}, 0xc469c4?)
        github.com/tendermint/[email protected]/state/execution.go:307 +0x51d
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x14}, {0xc0f483d990, 0x7}}, {0xc0f483d997, 0x9}, 0x1, 0xc469c4, {{0xc01771d2e0, ...}, ...}, ...}, ...)
        github.com/tendermint/[email protected]/state/execution.go:140 +0x171
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc0004b61c0, 0x0)
        github.com/tendermint/[email protected]/blockchain/v0/reactor.go:400 +0xbda
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
        github.com/tendermint/[email protected]/blockchain/v0/reactor.go:112 +0x7a

Dec 22 '23 05:12 ko0f

The same above also reproduces on a different machine.

Dec 22 '23 05:12 ko0f

Tried to build docker image from Dockerfile, same issue occurs when running.

Dec 22 '23 07:12 ko0f

osmosis osmosis copied to clipboard

[Bug]: "panic: unexpected EOF" upon startup

What happened?

Osmosis Version

How to reproduce?

osmosis
osmosis copied to clipboard