Panic during block production

[ilhanu]

Hi our Calamari Collator had the following panic on version 4.3.0, just updated to v4.6.0 since I saw most of the calamari services also upgrades.

After the upgrade, seeing this though: 2024-01-11 09:32:00 [Parachain] Logic error: Unexpected underflow in reducing consumer

The panic logs are below in the codeblocks, this was version v.4.3.0

2024-01-11 09:17:48 [Parachain] panicked at /home/builder/cargo/git/checkouts/nimbus-43d01ec6e06cfb96/bc3b008/pallets/author-inherent/src/lib.rs:137:13:
Block invalid, supplied author is not eligible.
2024-01-11 09:17:48 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-01-11 09:17:48 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-01-11 09:17:48 [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
    0: 0x49052d - <unknown>!rust_begin_unwind
    1: 0x44a813 - <unknown>!core::panicking::panic_fmt::h7a368385936888dc
    2: 0xdd4f0 - <unknown>!frame_support::storage::transactional::with_transaction::hdfc1bdd2efa31d9a
    3: 0x13ab0d - <unknown>!frame_support::dispatch_context::run_in_context::h607bfd1b757f5604
    4: 0x6ed39 - <unknown>!<calamari_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h6e865962d4cbe387
    5: 0x6afeb - <unknown>!<calamari_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::hecc2da664741150f
    6: 0x13613d - <unknown>!<sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply::hca207c993d312df2
    7: 0x3d1d3f - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::h2171a10fc98e6b2b
    8: 0x1c7adf - <unknown>!BlockBuilder_apply_extrinsic
. Dropping.
2024-01-11 09:17:48 [Parachain] 🎁 Prepared block for proposing at 5346747 (6 ms) [hash: 0xdbf6de8b190e1d176fcdb0ecace0e6628e0b9f2d69f82675118e4736718d09ac; parent_hash: 0xd851…3a29; extrinsics (6): [0x95e4…4038, 0x85cf…a62f, 0x6b04…49e3, 0xeeaa…16f3, 0xf28c…6dd7, 0x4b1b…a3b2]]
2024-01-11 09:17:48 [Parachain] 🔖 Sealed block for proposal at 5346747. Hash now 0xf0b8ee53a8b2d80de65528153ac79fc67ea4897698da07c3e1f1a69bbf71cad5, previously 0xdbf6de8b190e1d176fcdb0ecace0e6628e0b9f2d69f82675118e4736718d09ac.
2024-01-11 09:17:48 [Parachain] ✨ Imported #5346747 (0xf0b8…cad5)
2024-01-11 09:17:48 [Parachain] PoV size { header: 0.24609375kb, extrinsics: 10.4541015625kb, storage_proof: 345.759765625kb }
2024-01-11 09:17:48 [Parachain] Compressed PoV size: 238.716796875kb
2024-01-11 09:17:48 [Parachain] Produced proof-of-validity candidate. block_hash=0xf0b8ee53a8b2d80de65528153ac79fc67ea4897698da07c3e1f1a69bbf71cad5
2024-01-11 09:17:48 [Relaychain] Received finalized block via RPC: #21383195 (0x5bae…9981 -> 0x14e0…9711)
2024-01-11 09:17:52 [Parachain] 💤 Idle (39 peers), best: #5346746 (0xd851…3a29), finalized #5346746 (0xd851…3a29), ⬇ 1.6kiB/s ⬆ 2.0kiB/s
2024-01-11 09:17:52 [Relaychain] Received finalized block via RPC: #21383196 (0x14e0…9711 -> 0xc02e…3dab)
2024-01-11 09:17:54 [Relaychain] Received imported block via RPC: #21383199 (0x385e…8168 -> 0xad6a…8fdb)
2024-01-11 09:17:54 [Parachain] Starting collation. relay_parent=0xad6a8a841e01eddda073c86dedaf0c9d415aae765e19bc21c6b7670deea98fdb at=0xd851ba8a77b781fd72636490de642ff5ed27791590a808e8873277835f303a29
2024-01-11 09:17:54 [Parachain] 🙌 Starting consensus session on top of parent 0xd851ba8a77b781fd72636490de642ff5ed27791590a808e8873277835f303a29
2024-01-11 09:17:54 [Parachain] panicked at /home/builder/cargo/git/checkouts/nimbus-43d01ec6e06cfb96/bc3b008/pallets/author-inherent/src/lib.rs:137:13:
Block invalid, supplied author is not eligible.
2024-01-11 09:17:54 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-01-11 09:17:54 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-01-11 09:17:54 [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
    0: 0x49052d - <unknown>!rust_begin_unwind
    1: 0x44a813 - <unknown>!core::panicking::panic_fmt::h7a368385936888dc
    2: 0xdd4f0 - <unknown>!frame_support::storage::transactional::with_transaction::hdfc1bdd2efa31d9a
    3: 0x13ab0d - <unknown>!frame_support::dispatch_context::run_in_context::h607bfd1b757f5604
    4: 0x6ed39 - <unknown>!<calamari_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h6e865962d4cbe387
    5: 0x6afeb - <unknown>!<calamari_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::hecc2da664741150f
    6: 0x13613d - <unknown>!<sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply::hca207c993d312df2
    7: 0x3d1d3f - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::h2171a10fc98e6b2b
    8: 0x1c7adf - <unknown>!BlockBuilder_apply_extrinsic
. Dropping.
2024-01-11 09:17:54 [Parachain] 🎁 Prepared block for proposing at 5346747 (3 ms) [hash: 0x1703650f2e0913f3b84a2a18beef55725afefbf932ac5065fa4f94198525d4f4; parent_hash: 0xd851…3a29; extrinsics (6): [0x8d33…8005, 0x32b0…feff, 0x6b04…49e3, 0xeeaa…16f3, 0xf28c…6dd7, 0x4b1b…a3b2]]
2024-01-11 09:17:54 [Parachain] 🔖 Sealed block for proposal at 5346747. Hash now 0x451801afd6e2bf6417dc24e059140bbb2f39ca0fce7f035171be768f15963734, previously 0x1703650f2e0913f3b84a2a18beef55725afefbf932ac5065fa4f94198525d4f4.

[ferrell-code]

Thanks for the report!

Is this only happening when your collator binary version is 4.3.0?

Or do you continue to get this log once running 4.6.0. I don't see this log on our nodes but they were upgraded to 4.6.0 before runtime upgrade.

I believe the dependency bump was a breaking change, I will investigate.

[ilhanu]

Yes the error continues to happen also on v4.6.0 and is the exact same panic message as on the previous version.

I should add to the issue that I use the --relay-chain-rpc-urls flag that points the the WS-RPC of polkadot 1.5.0-a3dc2f15f23 relaychain node which is running locally. So I'm not running the embedded relay node.

[Dengjianping]

Can you tell me the name of you collator from here and collator account?

[ilhanu]

staker-space dmx4WhyUDhAjsMf1mRD55qApjxnqSXcSsmweHgcv8seGkrN4R

[Dengjianping]

If it's not fixed, please re-open it