Node Sync Error， ERROR runtime: Failed to check if a state snapshot exists err=STATE_SNAPSHOT_KEY

[mayong82]

2024-03-23T02:01:04.193141Z INFO neard: version="1.38.0" build="1.38.0" latest_protocol=65 2024-03-23T02:01:04.204584Z WARN neard: /data/nodedata/config.json: encountered unrecognised fields: store.state_snapshot_config.compaction_enabled, store.state_snapshot_compaction_enabled 2024-03-23T02:01:04.204604Z INFO config: Validating Config, extracted from config.json... 2024-03-23T02:01:04.214968Z WARN genesis: Skipped genesis validation 2024-03-23T02:01:04.214986Z WARN genesis: Skipped genesis validation 2024-03-23T02:01:04.214992Z INFO config: All validations have passed! 2024-03-23T02:01:04.223377Z INFO neard: Reset the config "/data/nodedata/log_config.json" because the config file doesn't exist. err=Os { code: 2, kind: NotFound, message: "No such file or directory" } 2024-03-23T02:01:04.224186Z WARN neard: /data/nodedata/config.json: encountered unrecognised fields: store.state_snapshot_config.compaction_enabled, store.state_snapshot_compaction_enabled 2024-03-23T02:01:04.224193Z INFO config: Validating Config, extracted from config.json... 2024-03-23T02:01:04.225449Z INFO near_o11y::reload: Updated the logging layer according to log_config.json 2024-03-23T02:01:04.226285Z INFO db_opener: Opening NodeStorage path="/data/nodedata/data" cold_path="none" 2024-03-23T02:01:04.228636Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-03-23T02:01:05.452289Z INFO db: Closed a RocksDB instance. num_instances=0 2024-03-23T02:01:05.452321Z INFO db_opener: The database exists. path=/data/nodedata/data 2024-03-23T02:01:05.452331Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-03-23T02:01:09.265739Z INFO db: Closed a RocksDB instance. num_instances=0 2024-03-23T02:01:09.265804Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-03-23T02:01:09.301735Z INFO db: Closed a RocksDB instance. num_instances=0 2024-03-23T02:01:09.301768Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-03-23T02:01:09.618297Z ERROR runtime: Failed to check if a state snapshot exists err=STATE_SNAPSHOT_KEY thread 'main' panicked at chain/client/src/client_actor.rs:2070:6: called Result::unwrap() on an Err value: Chain(DBNotFoundErr("epoch block: 8UWt3HW75CMAzXUTFtyAS4kCVy3zs6CcMh39RZLBJZuU")) stack backtrace: 0: rust_begin_unwind at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5 1: core::panicking::panic_fmt at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14 2: core::result::unwrap_failed at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1653:5 3: nearcore::start_with_config_and_synchronization 4: neard::cli::RunCmd::run::{{closure}} 5: tokio::task::local::LocalSet::run_until::{{closure}} 6: neard::cli::NeardCmd::parse_and_run 7: neard::main note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

Previously, an error occurred when running version 1.37. The board was upgraded to 1.37.3, and then the snapshot was downloaded again. At first, it ran normally, but after a period of time, this error persisted. Later, it was upgraded to 1.38.0 and the snapshot was downloaded again. At first, it ran normally, but later it continued to report this error. May I ask how to resolve this error;

[icculp]

I just ran into this on a freshly downloaded snap, it was syncing on 1.38.0 although extremely slowly, restarted to update to 1.38.1 and gave me this error

[Notation]

same issue, upgrading from 1.38.1 to 1.39.0

[mayong82]

Upgrading to 1.39.0 is still the same issue

2024-04-12T15:27:16.842076Z INFO neard: version="1.39.0" build="1.39.0" latest_protocol=66 2024-04-12T15:27:16.865264Z WARN neard: /data/nodedata/config.json: encountered unrecognised fields: store.state_snapshot_config.compaction_enabled, store.state_snapshot_compaction_enabled 2024-04-12T15:27:16.865292Z INFO config: Validating Config, extracted from config.json... 2024-04-12T15:27:16.875561Z WARN genesis: Skipped genesis validation 2024-04-12T15:27:16.875586Z WARN genesis: Skipped genesis validation 2024-04-12T15:27:16.875592Z INFO config: All validations have passed! 2024-04-12T15:27:16.884811Z INFO neard: Reset the config "/data/nodedata/log_config.json" because the config file doesn't exist. err=Os { code: 2, kind: NotFound, message: "No such file or directory" } 2024-04-12T15:27:16.885707Z WARN neard: /data/nodedata/config.json: encountered unrecognised fields: store.state_snapshot_config.compaction_enabled, store.state_snapshot_compaction_enabled 2024-04-12T15:27:16.885714Z INFO config: Validating Config, extracted from config.json... 2024-04-12T15:27:16.885759Z INFO near_o11y::reload: Updated the logging layer according to log_config.json 2024-04-12T15:27:16.886457Z INFO db_opener: Opening NodeStorage path="/data/nodedata/data" cold_path="none" 2024-04-12T15:27:16.889794Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-04-12T15:27:18.476178Z INFO db: Closed a RocksDB instance. num_instances=0 2024-04-12T15:27:18.476224Z INFO db_opener: The database exists. path=/data/nodedata/data 2024-04-12T15:27:18.476235Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-04-12T15:27:22.607699Z INFO db: Closed a RocksDB instance. num_instances=0 2024-04-12T15:27:22.607759Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-04-12T15:27:22.650835Z INFO db: Closed a RocksDB instance. num_instances=0 2024-04-12T15:27:22.650879Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-04-12T15:27:23.032174Z ERROR runtime: Failed to check if a state snapshot exists err=STATE_SNAPSHOT_KEY thread 'main' panicked at chain/client/src/client_actor.rs:2070:6: called Result::unwrap() on an Err value: Chain(DBNotFoundErr("epoch block: q4382BrxWdn7g81YXNExnc4cSZNK2ysQQkUnFVhHENg")) stack backtrace: 0: rust_begin_unwind at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5 1: core::panicking::panic_fmt at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14 2: core::result::unwrap_failed at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1653:5 3: nearcore::start_with_config_and_synchronization 4: neard::cli::RunCmd::run::{{closure}} 5: tokio::task::local::LocalSet::run_until::{{closure}} 6: neard::cli::NeardCmd::parse_and_run 7: neard::main note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

Who solved this problem and sought help ？

[mayong82]

Upgrading to 1.39.0 is still the same issue

2024-04-12T15:27:16.842076Z INFO neard: version="1.39.0" build="1.39.0" latest_protocol=66 2024-04-12T15:27:16.865264Z WARN neard: /data/nodedata/config.json: encountered unrecognised fields: store.state_snapshot_config.compaction_enabled, store.state_snapshot_compaction_enabled 2024-04-12T15:27:16.865292Z INFO config: Validating Config, extracted from config.json... 2024-04-12T15:27:16.875561Z WARN genesis: Skipped genesis validation 2024-04-12T15:27:16.875586Z WARN genesis: Skipped genesis validation 2024-04-12T15:27:16.875592Z INFO config: All validations have passed! 2024-04-12T15:27:16.884811Z INFO neard: Reset the config "/data/nodedata/log_config.json" because the config file doesn't exist. err=Os { code: 2, kind: NotFound, message: "No such file or directory" } 2024-04-12T15:27:16.885707Z WARN neard: /data/nodedata/config.json: encountered unrecognised fields: store.state_snapshot_config.compaction_enabled, store.state_snapshot_compaction_enabled 2024-04-12T15:27:16.885714Z INFO config: Validating Config, extracted from config.json... 2024-04-12T15:27:16.885759Z INFO near_o11y::reload: Updated the logging layer according to log_config.json 2024-04-12T15:27:16.886457Z INFO db_opener: Opening NodeStorage path="/data/nodedata/data" cold_path="none" 2024-04-12T15:27:16.889794Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-04-12T15:27:18.476178Z INFO db: Closed a RocksDB instance. num_instances=0 2024-04-12T15:27:18.476224Z INFO db_opener: The database exists. path=/data/nodedata/data 2024-04-12T15:27:18.476235Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-04-12T15:27:22.607699Z INFO db: Closed a RocksDB instance. num_instances=0 2024-04-12T15:27:22.607759Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-04-12T15:27:22.650835Z INFO db: Closed a RocksDB instance. num_instances=0 2024-04-12T15:27:22.650879Z INFO db: Opened a new RocksDB instance. num_instances=1 2024-04-12T15:27:23.032174Z ERROR runtime: Failed to check if a state snapshot exists err=STATE_SNAPSHOT_KEY thread 'main' panicked at chain/client/src/client_actor.rs:2070:6: called Result::unwrap() on an Err value: Chain(DBNotFoundErr("epoch block: q4382BrxWdn7g81YXNExnc4cSZNK2ysQQkUnFVhHENg")) stack backtrace: 0: rust_begin_unwind at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5 1: core::panicking::panic_fmt at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14 2: core::result::unwrap_failed at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1653:5 3: nearcore::start_with_config_and_synchronization 4: neard::cli::RunCmd::run::{{closure}} 5: tokio::task::local::LocalSet::run_until::{{closure}} 6: neard::cli::NeardCmd::parse_and_run 7: neard::main note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

Who solved this problem and sought help ？

[SKlarsen84]

I'm also seeing this problem on 1.39.0