Error with Collator - Binary v2.19.0-24c7810f200

[paradox-tt]

Dear Devs,

At 4:00 AM UK time our Collator encountered the following errors and lost sync. This block of errors is repeated multiple times.

Dec 20 04:00:00 Kurura karura[477345]: WASM backtrace:
Dec 20 04:00:00 Kurura karura[477345]: error while executing at wasm backtrace:
Dec 20 04:00:00 Kurura karura[477345]:     0: 0x5b9b16 - <unknown>!rust_begin_unwind
Dec 20 04:00:00 Kurura karura[477345]:     1: 0xf3bb - <unknown>!core::panicking::panic_fmt::h3e1dd3d08288569e
Dec 20 04:00:00 Kurura karura[477345]:     2: 0xf84d - <unknown>!core::result::unwrap_failed::h8b3db0f11171b57b
Dec 20 04:00:00 Kurura karura[477345]:     3: 0x1d911c - <unknown>!frame_support::storage::transactional::with_transaction::h6bffe5eefc30dcf5
Dec 20 04:00:00 Kurura karura[477345]:     4: 0x4be327 - <unknown>!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::h2661feb062feae85
Dec 20 04:00:00 Kurura karura[477345]:     5: 0x4d0ff2 - <unknown>!frame_support::dispatch_context::run_in_context::h7ccb3c09309e2669
Dec 20 04:00:00 Kurura karura[477345]:     6: 0x6ba5a - <unknown>!<karura_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h5c4424f54dba32d5
Dec 20 04:00:00 Kurura karura[477345]:     7: 0x63d48 - <unknown>!<karura_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h0dce70915fd712dd
Dec 20 04:00:00 Kurura karura[477345]:     8: 0x4c4190 - <unknown>!<sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply::hd5e2072e8fff9956
Dec 20 04:00:00 Kurura karura[477345]:     9: 0x369273 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::h9ea8f24cd3b68233
Dec 20 04:00:00 Kurura karura[477345]:    10: 0x38f297 - <unknown>!BlockBuilder_apply_extrinsic. Dropping.
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] panicked at /home/builder/cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-parachain-system-0.4.1/src/lib.rs:268:18:
Dec 20 04:00:00 Kurura karura[477345]: set_validation_data inherent needs to be present in every block!
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] Proposing failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed

We're building our own binaries, please let us know if there's any issues with the approach. Thus far Acala is operating fine.

[xlc]

what’s the commit you using and can you share bit more logs before the error?

[paradox-tt]

Here's a bit more detail before the error and we're using 2.19 commit 24c7810f200

Dec 20 03:59:48 Kurura karura[477345]: 2023-12-20 03:59:48 [Parachain] Starting collation. relay_parent=0x81f70140e85dd4b36ead510ac46b2ec763768ea171736f94c8faca418b712534 at=0x21a6dff64d3c8a228377ecac6fd01ac1448e5bb724a96caf769c0b5d3728a92f
Dec 20 03:59:49 Kurura karura[477345]: 2023-12-20 03:59:48 [Relaychain] ✨ Imported #21065049 (0xc323…3966)
Dec 20 03:59:49 Kurura karura[477345]: 2023-12-20 03:59:49 [Parachain] Starting collation. relay_parent=0xc3238f2f3271f76d417ce10d8e6976544228e20f6fd539326d9df014c0dd3966 at=0x21a6dff64d3c8a228377ecac6fd01ac1448e5bb724a96caf769c0b5d3728a92f
Dec 20 03:59:49 Kurura karura[477345]: 2023-12-20 03:59:49 [Relaychain] 💤 Idle (7 peers), best: #21065049 (0x81f7…2534), finalized #21065045 (0x45d2…16ee), ⬇ 92.3kiB/s ⬆ 172.5kiB/s
Dec 20 03:59:49 Kurura karura[477345]: 2023-12-20 03:59:49 [Parachain] ✨ Imported #5888895 (0x3be7…4bd5)
Dec 20 03:59:49 Kurura karura[477345]: 2023-12-20 03:59:49 [Parachain] ✨ Imported #5888895 (0x5107…0f1e)
Dec 20 03:59:52 Kurura karura[477345]: 2023-12-20 03:59:51 [Parachain] 💤 Idle (28 peers), best: #5888894 (0x21a6…a92f), finalized #5888893 (0xb0af…798f), ⬇ 11.6kiB/s ⬆ 190.9kiB/s
Dec 20 03:59:54 Kurura karura[477345]: 2023-12-20 03:59:54 [Relaychain] 💤 Idle (6 peers), best: #21065049 (0x81f7…2534), finalized #21065046 (0x77c8…2844), ⬇ 410.2kiB/s ⬆ 454.8kiB/s
Dec 20 03:59:54 Kurura karura[477345]: 2023-12-20 03:59:54 [Relaychain] ✨ Imported #21065050 (0xf736…e638)
Dec 20 03:59:57 Kurura karura[477345]: 2023-12-20 03:59:56 [Parachain] 💤 Idle (29 peers), best: #5888894 (0x21a6…a92f), finalized #5888893 (0xb0af…798f), ⬇ 1.6kiB/s ⬆ 1.5kiB/s
Dec 20 03:59:59 Kurura karura[477345]: 2023-12-20 03:59:59 [Relaychain] 💤 Idle (7 peers), best: #21065050 (0xf736…e638), finalized #21065047 (0x5eca…5937), ⬇ 149.5kiB/s ⬆ 156.8kiB/s
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Relaychain] ✨ Imported #21065051 (0x48df…6c46)
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] Starting collation. relay_parent=0x48dfadd63b138aa86d6afcbc4ea5c49a91815949d4fa6e49974f3b62216d6c46 at=0x51079c54942995fc3c840d7ad0ececaee77b2b07e16e23608d8ebffcd51b0f1e
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] 🙌 Starting consensus session on top of parent 0x51079c54942995fc3c840d7ad0ececaee77b2b07e16e23608d8ebffcd51b0f1e
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] panicked at /home/builder/cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-parachain-system-0.4.1/src/lib.rs:626:18:
Dec 20 04:00:00 Kurura karura[477345]: Invalid messaging state in relay chain state proof: RelayDispatchQueueRemainingCapacity(Proof)
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
Dec 20 04:00:00 Kurura karura[477345]: 2023-12-20 04:00:00 [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed

[xlc]

we wasn’t expect breaking change and will investigate. but can you upgrade to 2.23?

[paradox-tt]

Built binaries using cargo build --locked --features with-karura-runtime --profile production --bin acala thus far all is working well but I'm holding for observation for another 24hrs.

[Luca-Poggi]

We had the same identical issue with Karura collator v2.19.0. Updated both Acala and Karura to v2.23.0. Let's keeping monitored!

[xlc]

v2.19.0 is known that contains multiple issues. everyone should upgrade to v2.23.0