backfill sync failed err=stage encountered an error in block #23371839: validation error: mismatched block state root:

[molaco]

trafficstars

Describe the bug

I have no idea what happened. I go backfill sync error. I did a checksum and came back good.

Steps to reproduce

I have no idea how to reproduce this but here is my flake.

flake.nix

{ description = "rust";

inputs = { nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; old-python-nixpkgs.url = "github:nixos/nixpkgs/2030abed5863fc11eccac0735f27a0828376c84e"; };

outputs = { self, nixpkgs, ... }@inputs: let system = "x86_64-linux"; pkgs = nixpkgs.legacyPackages.${system};

in
{
  devShells.x86_64-linux.default = (import ./shell.nix {inherit pkgs; });
};

}

shell.nix

{ pkgs ? import {} }: let overrides = (builtins.fromTOML (builtins.readFile ./rust-toolchain.toml)); libPath = with pkgs; lib.makeLibraryPath [ # load external libraries that you need in your rust project here ]; in pkgs.mkShell rec { buildInputs = with pkgs; [ clang # Replace llvmPackages with llvmPackages_X, where X is the latest LLVM version (at the time of writing, 16) llvmPackages.bintools rustup cargo-nextest

  # lighthouse dependencies
  libgcc
  gnumake
  cmake
  pkg-config
  libclang

];
RUSTC_VERSION = overrides.toolchain.channel;
# https://github.com/rust-lang/rust-bindgen#environment-variables
LIBCLANG_PATH = pkgs.lib.makeLibraryPath [ pkgs.llvmPackages_latest.libclang.lib ];
shellHook = ''
  export PATH=$PATH:''${CARGO_HOME:-~/.cargo}/bin
  export PATH=$PATH:''${RUSTUP_HOME:-~/.rustup}/toolchains/$RUSTC_VERSION-x86_64-unknown-linux-gnu/bin/
  # export NIX_BUILD_SHELL=$(which zsh)
  '';
# Add precompiled library to rustc search path
RUSTFLAGS = (builtins.map (a: ''-L ${a}/lib'') [
  # add libraries here (e.g. pkgs.libvmi)
]);
LD_LIBRARY_PATH = libPath;
# Add glibc, clang, glib, and other headers to bindgen search path
BINDGEN_EXTRA_CLANG_ARGS =
# Includes normal include path
(builtins.map (a: ''-I"${a}/include"'') [
  # add dev libraries here (e.g. pkgs.libvmi.dev)
  pkgs.glibc.dev
])
# Includes with special directory paths
++ [
  ''-I"${pkgs.llvmPackages_latest.libclang.lib}/lib/clang/${pkgs.llvmPackages_latest.libclang.version}/include"''
  ''-I"${pkgs.glib.dev}/include/glib-2.0"''
  ''-I${pkgs.glib.out}/lib/glib-2.0/include/''
];

}

Node logs

2025-09-28T23:37:49.657746Z  INFO Status connected_peers=100 stage=MerkleUnwind checkpoint=23462279 target=23371839
2025-09-28T23:37:58.526256Z  INFO Received forkchoice updated message when syncing head_block_hash=0x954dc1d9efdd7e739d994de737f5bf7e84e4a8993340094d9ba6cb8c87ab846f safe_block_hash=0x9396c787403e15d46a1fd42fb719f2a8270b669e880eb28b3b6eab82f0d3b8d3 finalized_block_hash=0x5289056f9683ee695fb0d779f844c4de70c46fc02111ffaf8b6f0ee73fab39e4
2025-09-28T23:38:02.118336Z  INFO Received block from consensus engine number=23464820 hash=0x2494c01bdcd7373bde56772a211badf07f0a85f2eb12f6d0f181140ae5b3891c
2025-09-28T23:38:05.589060Z ERROR Unwinding{stage=MerkleUnwind}: Failed to verify block state root! 
Invalid state root error on stage verification!
This is an error that likely requires a report to the reth team with additional information.
Please include the following information in your report:
 * This error message
 * The state root of the block that was rejected
 * The output of `reth db stats --checksum` from the database that was being used. This will take a long time to run!
 * 50-100 lines of logs before and after the first occurrence of the log message with the state root of the block that was rejected.
 * The debug logs from __the same time period__. To find the default location for these logs, run:
   `reth --help | grep -A 4 'log.file.directory'`

Once you have this information, please submit a github issue at https://github.com/paradigmxyz/reth/issues/new
 target_block=23371839 got=0xb3623d46227e67075dfb40e381cdc059fd963c90dd71bec9c6b9547979ab9156 expected=SealedHeader { hash: OnceLock(0x1521dc111ce6a5a4e8f335dbf92ce1b6e4f02f63d7f664a703b5d7fa14df71b0), header: Header { parent_hash: 0x2eff6da2695173af5b5d3b4694f0a8ca7f8355e7e7607ec34bb4813c33475f64, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0xdadb0d80178819f2319190d340ce9a924f783711, state_root: 0xe1e5e127b950d03841aec107c8c97002ccfa55737d28b50f23442e9eebabadd8, transactions_root: 0xe9c39837f5e6c13445991f9ac0f48ec5e0733dd273df0a325fbd8c06649ecbc5, receipts_root: 0xc3f3be669fb60fcb0dd4de19d3104a01b47faaa567f0fa5469f6d189f9bc7d77, logs_bloom: 0x1ff9bf8dffe9ff3e7bbeb17ffbf9babfffff7c7cffedfbdf375fffdd56ffbdfffff9f93ff7fcdee5fffedffff52fdfdbeaf5d7f5fffffff89bfdfffdedbe65ffd3afedbefcefdf7f7b73dfbffecbde7bf9ecbfffa6fef9fe7efebffefcd97dfa5ff7fb7efefa6ff57cddfbc7ddeeefa7feb7f3efffde36f5f7fdf536bedffff78ff77fffbbef7f6977ff9d73b7b7deffeed7f3f767ed72bfdffec6ff7cbe6effcffedff7afbbef8e5775fff2fab7fff336e3bbff3afcff893ffe7f7a6ffffefffdbedffff7fb7fffaef9ffbfbb9ffdaf0775fff6fdef6fd341fff5ffdcefebfbeff9ecfeef3ffdfefdff7fbfbfdf99afffadffff77bcb6ff6ddecde7effd7dff, difficulty: 0, number: 23371839, gas_limit: 44999957, gas_used: 30351710, timestamp: 1757979887, extra_data: 0x4275696c6465724e65742028466c617368626f747329, mix_hash: 0xdd651cec9d8704fa8f07284785e93d34d918e1f8fbee820bd078bafa9b6c4b71, nonce: 0x0000000000000000, base_fee_per_gas: Some(206171238), withdrawals_root: Some(0x2a23eb000dfdf733f50711d9cf73389de8cc6e475c4c40a9e46d11fcde263b1b), blob_gas_used: Some(655360), excess_blob_gas: Some(131072), parent_beacon_block_root: Some(0x2d72862f388cc4e35329af2100d8ce399563a7e1c5e4891557ca96d3dcc29186), requests_hash: Some(0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) } }
2025-09-28T23:38:09.313949Z  WARN The database read transaction has been open for too long. Backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: start_thread
  18: __clone3
 open_duration=3341.991199829s self.txn_id=23286286
2025-09-28T23:38:09.315051Z ERROR backfill sync failed err=stage encountered an error in block #23371839: validation error: mismatched block state root: got 0xb3623d46227e67075dfb40e381cdc059fd963c90dd71bec9c6b9547979ab9156, expected 0xe1e5e127b950d03841aec107c8c97002ccfa55737d28b50f23442e9eebabadd8
2025-09-28T23:38:09.320376Z ERROR Fatal error in consensus engine
2025-09-28T23:38:09.325308Z ERROR shutting down due to error
Error: Fatal error in consensus engine

Location:
    /home/molaco/Documents/rust/reth/crates/node/builder/src/launch/engine.rs:300:43

Platform(s)

Linux (x86)

Container Type

Other

What version/commit are you on?

reth-ethereum-cli Version: 1.8.1 Commit SHA: 48b725aec24ba5a702a5c875e6745b835248b17f Build Timestamp: 1980-01-01T00:00:00.000000000Z Build Features: jemalloc Build Profile: release

What database version are you on?

Lighthouse v7.0.1 BLS library: blst-modern BLS hardware acceleration: true SHA256 hardware acceleration: true Allocator: jemalloc (4K) Profile: release Specs: mainnet (true), minimal (false), gnosis (true)

Which chain / network are you on?

Ethereum mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

nothing just archive node

If you've built Reth from source, provide the full command you used

cargo install --path bin/reth --locked --force

Code of Conduct

• [x] I agree to follow the Code of Conduct