"A stage encountered an irrecoverable error."

[Gaon3]

Describe the bug

Node completed sync yesterday and ran for some time without troubles when this error happened.

root@Ubuntu-2204-jammy-amd64-base ~/.cache/reth/logs # reth node --log.persistent --log.filter debug
2023-04-28T10:56:53.772440Z  INFO reth 0.1.0 starting
2023-04-28T10:56:53.772663Z  INFO Configuration loaded path=/root/.config/reth/mainnet/reth.toml
2023-04-28T10:56:53.772682Z  INFO Opening database path=/root/.local/share/reth/mainnet/db
2023-04-28T10:56:53.773201Z  INFO Database opened
2023-04-28T10:56:53.809925Z  INFO Transaction pool initialized
2023-04-28T10:56:53.809944Z  INFO Connecting to P2P network
2023-04-28T10:56:53.935300Z  INFO Connected to P2P network peer_id=0xcd10…5dcb local_addr=0.0.0.0:30303
2023-04-28T10:56:53.935455Z  INFO Consensus engine initialized
2023-04-28T10:56:53.935461Z  INFO Engine API handler initialized
2023-04-28T10:56:53.935548Z  INFO Started RPC server
2023-04-28T10:56:53.935574Z  INFO Started Auth server
2023-04-28T10:56:53.935578Z  INFO Starting consensus engine
2023-04-28T10:56:53.936111Z  INFO Status connected_peers=0 stage=None checkpoint=0
2023-04-28T10:56:54.964318Z  INFO Peer connected connected_peers=1 peer_id=0x775e…7956 best_block=0x5fa5…62b3
2023-04-28T10:56:55.031965Z  INFO Peer connected connected_peers=2 peer_id=0xb411…ff60 best_block=0x998f…caad
2023-04-28T10:56:55.172506Z  INFO Peer connected connected_peers=3 peer_id=0x39a3…8757 best_block=0x5fa5…62b3
2023-04-28T10:57:03.582842Z  INFO Executing stage stage=Headers from=17140946
2023-04-28T10:57:03.941597Z  WARN Peer disconnected. connected_peers=2 peer_id=0xb411…ff60 reason=None
2023-04-28T10:57:04.039267Z  INFO Stage finished executing stage=Headers checkpoint=17141009
2023-04-28T10:57:04.039383Z  INFO Executing stage stage=TotalDifficulty from=17140946
2023-04-28T10:57:04.039472Z  INFO Stage finished executing stage=TotalDifficulty checkpoint=17141009
2023-04-28T10:57:04.039550Z  INFO Executing stage stage=Bodies from=17140946
2023-04-28T10:57:04.998170Z  INFO Stage finished executing stage=Bodies checkpoint=17141009
2023-04-28T10:57:05.103544Z  INFO Executing stage stage=SenderRecovery from=17140946
2023-04-28T10:57:05.130041Z  INFO Stage finished executing stage=SenderRecovery checkpoint=17141009
2023-04-28T10:57:05.130135Z  INFO Executing stage stage=Execution from=17140930
2023-04-28T10:57:05.190268Z  INFO Wrote network peers to file peers_file="/root/.local/share/reth/net/mainnet/known-peers.json"
Error: A stage encountered an irrecoverable error.

Caused by:
   0: Stage encountered a execution error in block 17140931: EVM reported invalid transaction (0x6681c7360b79d05fe9bfba3a22b02ff612fb5abf07f233bb441c791527d6c5ab): Transaction(NonceTooHigh { tx: 2548, state: 2547 }).
   1: EVM reported invalid transaction (0x6681c7360b79d05fe9bfba3a22b02ff612fb5abf07f233bb441c791527d6c5ab): Transaction(NonceTooHigh { tx: 2548, state: 2547 })

Location:
    bin/reth/src/node/mod.rs:380:9

It also happened like 5 days ago (but with a different error message) when I completed a previous sync, so it was not added in a recent commit.

I can not attach the logs because despite running --log.presistent, nothing gets written inside reth.log.

Steps to reproduce

Complete a sync, wait.

Node logs

No response

Platform(s)

Linux (x86)

What version/commit are you on?

eb37bc7b37b853d9bd11b98ea78652e575fd9466

Code of Conduct

• [X] I agree to follow the Code of Conduct

[rakita]

The exact block got reorged from etherscan, so it is probably something related.

Another thing is that: Executing stage stage=Headers from=17140946 header is at the block ...946 While Execution starts from Executing stage stage=Execution from=17140930 is at ...930

maybe target just got moved if new FCU is received from the engine (wild guess)

[gakonst]

@Gaon3 is this still a problem?

[onbjerg]

Ping @Gaon3

[gakonst]

Bump @Gaon3! Have you re-encountered this?

[gakonst]

Bump @Gaon3 - closing optimistically, please re-open if still present.