nimbus-eth1 icon indicating copy to clipboard operation
nimbus-eth1 copied to clipboard

Published 20 hours ago • verified publisher icon status-im

Sync assertion / crash on invalid block

Open arnetheduck opened this issue 10 months ago • 4 comments

On the forked-layers branch (merged with master):

...skipping...
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(129) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(271) _ZN13blocks_staged18blocksStagedImportE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(361) _ZN18blocksStagedImport18blocksStagedImportE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_unproc.nim(65) _ZN13blocks_unproc18blocksUnprocCommitE6CtxRefIN11worker_desc13BeaconCtxDataEE4uInt6uInt646uInt64
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_unproc.nim(73) _ZN13blocks_unproc18blocksUnprocCommitE6CtxRefIN11worker_desc13BeaconCtxDataEE4uInt6uInt646uInt64
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/std/assertions.nim(41) _ZN10assertions16failedAssertImplE6string
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/std/assertions.nim(36) _ZN10assertions11raiseAssertE6string
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(53) _ZN10assertions8sysFatalE8typeDescIN10exceptions15AssertionDefectEE6string
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/excpt.nim(345) _ZN6system18rawWriteStackTraceE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) mainJan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(102) _ZN6worker9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(129) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(271) _ZN13blocks_staged18blocksStagedImportE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(369) _ZN18blocksStagedImport18blocksStagedImportE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(102) _ZN6worker9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(129) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(271) _ZN13blocks_staged18blocksStagedImportE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(370) _ZN18blocksStagedImport18blocksStagedImportE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(102) _ZN6worker9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(127) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(102) _ZN6worker9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(105) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(102) _ZN6worker9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(106) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(38) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(242) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(243) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: Error: unhandled exception: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_unproc.nim(72, 3) `ctx.blk.unprocessed.merge(rMinPt, rMaxPt) == rMaxPt - rMinPt + 1`  [AssertionDefect]

INF 2025-01-30 22:11:20.450+01:00 Importing blocks                           topics="beacon sync" iv=#21013593..#21014200 nBlocks=608 base=#21013464 head=#21013592 target=#21739790
DBG 2025-01-30 22:11:21.027+01:00 wrong state root in block                  blockNumber=21013593 blockHash=0xe2d459fd8c2c972391da8cddfbac19f31413d779ba1b1c73c31bcabd7fd058aa parentHash=0x85cb24b00ed4204fe042865c4d70c8b6a86d6c212800075cba99e4be5f83ff62 expected=0x2db6858070460c80b6c9802f5dffca67bdb0ecfa184b279d71a1ca1ce2ca48aa actual=0x1159a10bc4fa3d2e13c4d5335aa382c7f9b6b509201c477ede5ab7da07424730 arrivedFrom=0x6284332f5b991dcac7dfc12e465dca8b7829a1e99e1458d2d241300091d63690
WRN 2025-01-30 22:11:21.027+01:00 RunDaemon: import block error              topics="beacon sync" n=0 iv=#21013593..#21014200 B=#21013464 L=#21013592 nthBn=#21013593 nthHash=e2d459d058aa error="stateRoot mismatch, expect: 0x2db6858070460c80b6c9802f5dffca67bdb0ecfa184b279d71a1ca1ce2ca48aa, got: 0x1159a10bc4fa3d2e13c4d5335aa382c7f9b6b509201c477ede5ab7da07424730"

arnetheduck avatar Jan 30 '25 21:01 arnetheduck

Is there a way to reproduce that? As a side note, there are some important syncer changes missing on this branch. The only changes against the current syncer source code tree that make currently sense to me are with the worker/db.nim file.

There is a potential problem with the error recovery after an import error (when there was an internal FC module reorg) which I will fix.

There is also a quick abort (for this sync scrum) useful when one can assume that the current scrum/sprint will fail at most or all blocks to be imported as it seems the case here. This will be updated when the forked-layers branch is merged and the return code messages are stable. This mechanism is available at other places already.

mjfh avatar Jan 31 '25 10:01 mjfh

Is there a way to reproduce that?

Not sure, it happened a few hours after starting my mainnet node from an snapshot db (from November)..

Of course, the fact that the block failed to apply is also an issue to be investigated, but that's a separate problem.

arnetheduck avatar Jan 31 '25 10:01 arnetheduck

What seemed to have happened was that the FC module switched to another canonical branch shortly before the syncer started importing (was the first block of the current batch.)

The assert came from a (reproducible) bookkeeping error when handling repeated import errors.

mjfh avatar Jan 31 '25 11:01 mjfh

What seemed to have happened was that the FC module switched to another canonical branch shortly before

to be clear, this is normal behavior - ie FC can switch to a different branch at any time and the syncer must make no assumptions about it.

arnetheduck avatar Jan 31 '25 14:01 arnetheduck