oasis-core icon indicating copy to clipboard operation
oasis-core copied to clipboard
verified publisher icon oasisprotocol

consensus/cometbft: Panic during termination

Open gw0 opened this issue 3 months ago • 0 comments

SUMMARY

I noticed a panic during termination of a consensus-only Oasis-node on Testnet when syncing from genesis and with pruning enabled.

ISSUE TYPE
  • Bug Report
COMPONENT NAME

consensus/cometbft

OASIS NODE VERSION
Software version: 25.5
Consensus:
  Consensus protocol version: 7.0.0
Runtime:
  Host protocol version:      5.1.0
  Committee protocol version: 5.0.0
Go toolchain version: 1.24.5
OS / ENVIRONMENT

Docker.

STEPS TO REPRODUCE

Terminate during genesis sync with pruning enabled.

ACTUAL RESULTS
Terminating processes...
{"caller":"indexer_service.go:97","height":28190272,"level":"info","module":"cometbft:txindex","msg":"indexed block events","ts":"2025-09-09T11:55:59.411248213Z"}
{"caller":"service_manager.go:60","level":"info","module":"oasis-node","msg":"user requested termination","ts":"2025-09-09T11:55:59.411666535Z"}
{"caller":"service_manager.go:66","level":"debug","module":"oasis-node","msg":"stopping services","ts":"2025-09-09T11:55:59.412050792Z"}
{"caller":"service_manager.go:69","level":"debug","module":"oasis-node","msg":"stopping service","svc":"metrics","ts":"2025-09-09T11:55:59.412059518Z"}
{"caller":"service_manager.go:69","level":"debug","module":"oasis-node","msg":"stopping service","svc":"pprof","ts":"2025-09-09T11:55:59.41259171Z"}
{"caller":"service_manager.go:69","level":"debug","module":"oasis-node","msg":"stopping service","svc":"grpc/internal","ts":"2025-09-09T11:55:59.412802282Z"}
{"caller":"component.go:34","level":"info","module":"grpc","msg":"[core] [Server #1 ListenSocket #2]ListenSocket deleted\n","ts":"2025-09-09T11:55:59.413992154Z"}
{"caller":"state.go:609","level":"debug","module":"abci-mux/state","msg":"state pruner is terminating","ts":"2025-09-09T11:55:59.414350996Z"}
{"block_height":28190272,"caller":"mux.go:575","hash":"02512f9c83008961fd7cf6b1897fd5c95897b4db9b9c9857a0b3ddd10050e804","level":"debug","module":"abci-mux","msg":"BeginBlock","req":{"hash":"AlEvnIMAiWH9fPaxiX/VyViXtNubnJhXoLPd0QBQ6AQ=","
header":{"version":{"block":11,"app":30064771072},"chain_id":"0b91b8e4e44b2003a7c5e23ddadb5e14ef5345c0ebcb3ddcae","height":28190273,"time":"2025-08-29T20:52:52.118679032Z","last_block_id":{"hash":"BS6obTVhwuGjouQEwLzmaNGLUDQvEKIYiyfOMGSNM
ss=","part_set_header":{"total":1,"hash":"FB6vJTxDZc3VVDgNvcotqYbTlM8c3NhsHd4wwuxJ58k="}},"last_commit_hash":"w5Vk6vZwziMuliNRBYmSvJbohnmJNUsHGYWQsowl/6I=","data_hash":"+sV7sEe3YCB/2CMSfjZObMcNJYPhHzpH2vo0tmi0AbE=","validators_hash":"qi+4
BMOMDWzHKHryQogTMgTWjql4aKwXvUtVcKgZE7M=","next_validators_hash":"qi+4BMOMDWzHKHryQogTMgTWjql4aKwXvUtVcKgZE7M=","consensus_hash":"KU2PvQuUt2en66mEDymaNYbaf+a13q07fuy6GTxAD5M=","app_hash":"yaeegr3+Xp5wVoKsnWtJJMt6CquXwRLmtJH2nxa/hXc=","las
t_results_hash":"oaqo7lIqYgx71CWlrEH49bwIqtYSdiWebVcfVYYt0r4=","evidence_hash":"47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=","proposer_address":"rNZHfDkyWHcQqQ6EYZ5PsF3xp3M="},"last_commit_info":{"votes":[{"validator":{"address":"Gh6prLo
vvnagTR1yE8O4cgRZxVc=","power":723695445358434},"signed_last_block":true},{"validator":{"address":"rNZHfDkyWHcQqQ6EYZ5PsF3xp3M=","power":714242379195173},"signed_last_block":true},{"validator":{"address":"kAV4DX/XoUyq71vc0fbdAdxb0UE=","po
wer":704736122905820},"signed_last_block":true},{"validator":{"address":"GFt/wA+UZd0U1JDLuHJm1Q9YtHo=","power":2875547945861},"signed_last_block":true},{"validator":{"address":"LUJk1+htz86GlQ8o59zMYCE8Axc=","power":2459858750407},"signed_
last_block":true},{"validator":{"address":"acLXW8wkcHmuTuhcNh8vTLzrQ8k=","power":879023266291},"signed_last_block":true},{"validator":{"address":"9WtqCdZtFffGugd1M02UGlxNfmI=","power":838549999373},"signed_last_block":true},{"validator":{
"address":"Mi71Wy7YGY0+XrtptnpQqqzyTT8=","power":806656862001},"signed_last_block":true},{"validator":{"address":"FHbuz3L4PLFy4VzIAgtDfW79Ryg=","power":761273332593},"signed_last_block":true},{"validator":{"address":"RKiB+059yU0M6kp8B4mIW
MH36AU=","power":722784366304},"signed_last_block":true},{"validator":{"address":"bJk30p7Jb4WpAcZapzTaVj2VFog=","power":715564786200},"signed_last_block":true},{"validator":{"address":"uIIjHUymW8bm3S9htR6l9pBMfVo=","power":702812112322},"
signed_last_block":true},{"validator":{"address":"nG+onM4vXvc9iiDqSA/ZPJhdMZ4=","power":700495941388},"signed_last_block":true},{"validator":{"address":"Bc8T6ZYRTHM1oAiwojdkdpgwP3w=","power":642819660986},"signed_last_block":true},{"valid
ator":{"address":"IHjdfLI1KbrP6MQXM23zRgJdO00=","power":623346833530},"signed_last_block":true},{"validator":{"address":"4R0ML0nH9qIO9oE3AOA7i1aolxI=","power":507818690210},"signed_last_block":true},{"validator":{"address":"1QQyeWsqZvwaIu
2NgYNDndRxSLw=","power":370772470520},"signed_last_block":true},{"validator":{"address":"I3FMxeWT28KZj4amxNU11QEjblc=","power":369407857142},"signed_last_block":true},{"validator":{"address":"KCSd7yylCB6Dqa9Lqx+efQvlUsU=","power":27658332
1376},"signed_last_block":true},{"validator":{"address":"3KRvutTrwx2YIPTFrQPA0GV9o0s=","power":246527817418},"signed_last_block":true},{"validator":{"address":"CDfdbvLlT7oqVuZhe0ywc7Nxf9o=","power":150304845104},"signed_last_block":true},
{"validator":{"address":"GDT/YDtbyJs07piHI+XYd6jW2VA=","power":131557904662},"signed_last_block":true},{"validator":{"address":"r4hfwzQh0bUH/Dq8rbYoqfRPCds=","power":121207127230},"signed_last_block":true},{"validator":{"address":"t0v1ULs
3DpjNL2tsXAUIcvovdEg=","power":100499251513},"signed_last_block":true},{"validator":{"address":"lBnrd2UqJ3Vd37fseEYNMJP7oMs=","power":90145344914},"signed_last_block":true},{"validator":{"address":"Kg2e1LpLJVCAuD1Ec+5DM5Cf7iM=","power":85
016866287},"signed_last_block":true},{"validator":{"address":"/szIE9speVTrd9e8B0ZGCMsTlfg=","power":84675798210},"signed_last_block":true},{"validator":{"address":"blmDrkyxT2sy45IcuqdqpGn4zcc=","power":79866024022},"signed_last_block":tru
e},{"validator":{"address":"qAdtK55jcqFnV/GfqMo02iN5AOA=","power":75601354063},"signed_last_block":true},{"validator":{"address":"+wCmRvAou1WvtIHl+1HqSvLg6Xc=","power":74605297078},"signed_last_block":true},{"validator":{"address":"sdF1KQ
m510D+7wpRFW77kEIhZLY=","power":72756525474},"signed_last_block":true},{"validator":{"address":"0imQprislW8xH2ZdFa623hdQPFA=","power":71712079744},"signed_last_block":true},{"validator":{"address":"N1ugQf62r3vdtfW4cRoIkuEoJlU=","power":67
038342466},"signed_last_block":true},{"validator":{"address":"wt/X6xKTjAIocaSHqTsjlCU0vak=","power":60740414190}},{"validator":{"address":"/5btCAJuDCGdVPqmaM1jBQFzgmE=","power":57530854845},"signed_last_block":true},{"validator":{"address
":"Lma9MneOiqZAuOCrzX16WfFEn9g=","power":46021976339},"signed_last_block":true},{"validator":{"address":"iHuABd4Dkw2fdWi5QcRnpE4gq3s=","power":27928373292},"signed_last_block":true},{"validator":{"address":"q1aB0ciraf2vtzmxc7k39JBHC1o=","
power":27355347629},"signed_last_block":true},{"validator":{"address":"FIaLHcvM7Ahkyzdn5wg9dFsNSnE=","power":27353469972},"signed_last_block":true},{"validator":{"address":"2mdMa16IlhGvLoEmAaDyU7wtqqA=","power":26419878830},"signed_last_b
lock":true},{"validator":{"address":"VwvPPlq6gfC1qHJs7yne3FQgzWU=","power":19068611620},"signed_last_block":true},{"validator":{"address":"urwvzuJcuvdxeo/dm+LF8L2VYr8=","power":18270175103},"signed_last_block":true},{"validator":{"address
":"bb9XnZLBwbnrWPVCWgYeqWYPO+Q=","power":12683586688},"signed_last_block":true}]},"byzantine_validators":null},"ts":"2025-09-09T11:55:59.414404688Z"}
{"caller":"service_manager.go:69","level":"debug","module":"oasis-node","msg":"stopping service","svc":"worker p2p","ts":"2025-09-09T11:55:59.414490936Z"}
panic: mux: can't get current epoch in BeginBlock: failed to get future epoch for height 28190273: unavailable/corrupted state: context canceled

goroutine 20909 [running]:
github.com/oasisprotocol/oasis-core/go/consensus/cometbft/abci.(*abciMux).BeginBlock(_, {{0xc0108ccec0, 0x20, 0x20}, {{0xb, 0x700000000}, {0xc01b3b15c0, 0x32}, 0x1ae2641, {0x712e5f8, ...}, ...}, ...})
        github.com/oasisprotocol/oasis-core/go/consensus/cometbft/abci/mux.go:603 +0xd25
github.com/cometbft/cometbft/abci/client.(*localClient).BeginBlockSync(_, {{0xc0108ccec0, 0x20, 0x20}, {{0xb, 0x700000000}, {0xc01b3b15c0, 0x32}, 0x1ae2641, {0x712e5f8, ...}, ...}, ...})
        github.com/cometbft/[email protected]/abci/client/local_client.go:283 +0xf5
github.com/cometbft/cometbft/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc0108ccec0, 0x20, 0x20}, {{0xb, 0x700000000}, {0xc01b3b15c0, 0x32}, 0x1ae2641, {0x712e5f8, ...}, ...}, ...})
        github.com/cometbft/[email protected]/proxy/app_conn.go:101 +0x175
github.com/cometbft/cometbft/state.execBlockOnProxyApp({0x25d9d80, 0xc0008c8e40}, {0x25eca20, 0xc0057d4048}, 0xc01e8a7680, {0x25f0928, 0xc01b19a4b0}, 0x10ede81)
        github.com/cometbft/[email protected]/state/execution.go:374 +0x4fd
github.com/cometbft/cometbft/state.(*BlockExecutor).applyBlock(_, {{{0xb, 0x700000000}, {0xc0064ed248, 0x7}}, {0xc012dde100, 0x32}, 0x10ede81, 0x1ae2640, {{0xc01b424200, ...}, ...}, ...}, ...)
        github.com/cometbft/[email protected]/state/execution.go:208 +0x10e
github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyVerifiedBlock(...)
        github.com/cometbft/[email protected]/state/execution.go:186
github.com/cometbft/cometbft/blocksync.(*Reactor).poolRoutine(0xc02099fc00, 0x0)
        github.com/cometbft/[email protected]/blocksync/reactor.go:426 +0xba6
created by github.com/cometbft/cometbft/blocksync.(*Reactor).OnStart in goroutine 1
        github.com/cometbft/[email protected]/blocksync/reactor.go:136 +0x6b
EXPECTED RESULTS

Clear termination.

gw0 avatar Sep 09 '25 12:09 gw0