beta.5 rolling restart after sync

[mrabino1]

Describe the bug

rolling restart after sync with beta.5

Steps to reproduce

sync. restart. then it starts rolling restart

Node logs

2024-04-13T08:42:02.076309Z  INFO Database opened 
2024-04-13T08:42:02.076381Z  INFO 
Pre-merge hard forks (block based): 
- Frontier                         @0 
- Homestead                        @1150000 
- Dao                              @1920000 
- Tangerine                        @2463000 
- SpuriousDragon                   @2675000 
- Byzantium                        @4370000 
- Constantinople                   @7280000 
- Petersburg                       @7280000 
- Istanbul                         @9069000 
- MuirGlacier                      @9200000 
- Berlin                           @12244000 
- London                           @12965000 
- ArrowGlacier                     @13773000 
- GrayGlacier                      @15050000 
Merge hard forks: 
- Paris                            @58750000000000000000000 (network is known to be merged) 
Post-merge hard forks (timestamp based): 
- Shanghai                         @1681338455 
- Cancun                           @1710338135 
2024-04-13T08:42:02.076612Z  WARN Could not find block or tx number on a range request segment=Headers number=19645374 
2024-04-13T08:42:02.077033Z ERROR shutting down due to error 
Error: not able to find Headers static file for block number 19645374 
Location: 
    /project/crates/node-core/src/node_config.rs:447:20 
reth.service: Main process exited, code=exited, status=1/FAILURE 
reth.service: Failed with result 'exit-code'. 
reth.service: Scheduled restart job, restart counter is at 136. 
Stopped Reth Execution Client (Mainnet). 
Started Reth Execution Client (Mainnet).

Platform(s)

Linux (x86)

What version/commit are you on?

beta.5

What database version are you on?

beta.5

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

n/a

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• [X] I agree to follow the Code of Conduct

[mrabino1]

note: this started to occur after a power outage.. so I suspect something wasnt written to disk. do I need to resync again?

[mrabino1]

here is the reth.log

2024-04-13T08:28:51.145719Z DEBUG providers::db: Inserted block block_number=19645468 actions=[(InsertCanonicalHeaders, 33.739µs), (InsertHeaders, 13.905µs), (InsertHeaderNumbers, 30.445µs), (GetParentTD, 1.156µs), (InsertHeaderTerminalDifficulties, 7.423µs), (GetNextTxNum, 5.64µs), (InsertTransactionSenders, 0ns), (InsertTransactions, 1.661422ms), (InsertTransactionHashNumbers, 67.156448ms), (InsertBlockWithdrawals, 24.535µs), (InsertBlockBodyIndices, 17.323µs), (InsertTransactionBlocks, 18.746µs)]
2024-04-13T08:28:51.324700Z DEBUG providers::db: Appended blocks range=19645468..=19645468 actions=[(InsertBlock, 69.040835ms), (InsertState, 16.697603ms), (InsertHashes, 88.507531ms), (InsertHistoryIndices, 73.733485ms), (UpdatePipelineStages, 21.568µs)]
2024-04-13T08:28:51.387633Z DEBUG storage::db::mdbx: Commit total_duration=62.915664ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 15, audit: 0, write: 0, sync: 4104, ending: 0, whole: 4119, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-13T08:28:51.387665Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 18.875µs), (FindCanonicalHeader, 10.555µs), (SplitChain, 2.183µs), (SplitChainForks, 1.768µs), (MergeAllChains, 1.066µs), (UpdateCanonicalIndex, 4.629µs), (RetrieveStateTrieUpdates, 2.240315ms), (CommitCanonicalChainToDatabase, 310.97954ms)]
2024-04-13T08:28:51.387699Z DEBUG consensus::engine: Canonicalized new head hash=0x4f7a1f4b231b329521f684d5efa2f1da3ef0531eb54568185f79b5a6d1997e7c number=19645468
2024-04-13T08:28:51.387738Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-13T08:28:51.387826Z  INFO reth_node_core::events::node: Canonical chain committed number=19645468 hash=0x4f7a1f4b231b329521f684d5efa2f1da3ef0531eb54568185f79b5a6d1997e7c elapsed=313.300205ms
2024-04-13T08:29:01.262052Z DEBUG blockchain_tree: Appending block to canonical chain head=0xf7b7d845454ffa52e0fd61fd135df3c7c89a8237aca32790dc6e76f7abb6a195 parent=(19645468, 0x4f7a1f4b231b329521f684d5efa2f1da3ef0531eb54568185f79b5a6d1997e7c)
2024-04-13T08:29:01.349298Z DEBUG evm: Execution time evm_transact=85.044446ms apply_state=470.114µs apply_post_state=73.567µs merge_transitions=362.434µs receipt_root=866.545µs
2024-04-13T08:29:01.350263Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=300
2024-04-13T08:29:01.995837Z DEBUG blockchain_tree::chain: Validated state root number=19645469 hash=0xf7b7d845454ffa52e0fd61fd135df3c7c89a8237aca32790dc6e76f7abb6a195 elapsed=646.524174ms
2024-04-13T08:29:01.995944Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-13T08:29:01.995971Z  INFO reth_node_core::events::node: Block added to canonical chain number=19645469 hash=0xf7b7d845454ffa52e0fd61fd135df3c7c89a8237aca32790dc6e76f7abb6a195 peers=100 txs=173 mgas=10.371 full=34.6% base_fee=13.41gwei blobs=0 excess_blobs=0 elapsed=736.05ms
2024-04-13T08:29:02.177381Z DEBUG blockchain_tree: Committing new canonical chain: [(19645469, 0xf7b7d845454ffa52e0fd61fd135df3c7c89a8237aca32790dc6e76f7abb6a195)]
2024-04-13T08:29:02.179186Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19645469, 0xf7b7d845454ffa52e0fd61fd135df3c7c89a8237aca32790dc6e76f7abb6a195)]
2024-04-13T08:29:02.232410Z DEBUG providers::db: Inserted block block_number=19645469 actions=[(InsertCanonicalHeaders, 46.922µs), (InsertHeaders, 19.098µs), (InsertHeaderNumbers, 43.137µs), (GetParentTD, 1.531µs), (InsertHeaderTerminalDifficulties, 12.216µs), (GetNextTxNum, 6.566µs), (InsertTransactionSenders, 0ns), (InsertTransactions, 1.052734ms), (InsertTransactionHashNumbers, 51.899846ms), (InsertBlockWithdrawals, 26.499µs), (InsertBlockBodyIndices, 23.222µs), (InsertTransactionBlocks, 20.585µs)]
2024-04-13T08:29:02.351136Z DEBUG providers::db: Appended blocks range=19645469..=19645469 actions=[(InsertBlock, 53.209432ms), (InsertState, 12.966207ms), (InsertHashes, 59.709596ms), (InsertHistoryIndices, 46.007249ms), (UpdatePipelineStages, 22.692µs)]
2024-04-13T08:29:02.389950Z DEBUG storage::db::mdbx: Commit total_duration=38.796096ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 11, audit: 0, write: 0, sync: 2528, ending: 0, whole: 2541, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-13T08:29:02.389978Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 19.935µs), (FindCanonicalHeader, 16.612µs), (SplitChain, 1.802µs), (SplitChainForks, 1.302µs), (MergeAllChains, 855ns), (UpdateCanonicalIndex, 3.502µs), (RetrieveStateTrieUpdates, 1.814985ms), (CommitCanonicalChainToDatabase, 210.776707ms)]
2024-04-13T08:29:02.390008Z DEBUG consensus::engine: Canonicalized new head hash=0xf7b7d845454ffa52e0fd61fd135df3c7c89a8237aca32790dc6e76f7abb6a195 number=19645469
2024-04-13T08:29:02.390048Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-13T08:29:02.390088Z  INFO reth_node_core::events::node: Canonical chain committed number=19645469 hash=0xf7b7d845454ffa52e0fd61fd135df3c7c89a8237aca32790dc6e76f7abb6a195 elapsed=212.673391ms
2024-04-13T08:29:45.881379Z  INFO reth::cli: Starting reth version="0.2.0-beta.5 (54f75cd)"
2024-04-13T08:29:45.882471Z  INFO reth::cli: Opening database path="/home/mr/reth_storage/db"
2024-04-13T08:29:45.919181Z  INFO reth::cli: Configuration loaded path="/home/mr/reth_storage/reth.toml"
2024-04-13T08:29:45.940660Z  INFO reth::cli: Database opened
2024-04-13T08:29:45.941070Z DEBUG reth::cli: Initializing genesis chain=mainnet genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
2024-04-13T08:29:45.943206Z DEBUG reth_node_core::init: Genesis already written, skipping.
2024-04-13T08:29:45.943575Z  INFO reth::cli: 
Pre-merge hard forks (block based):
- Frontier                         @0
- Homestead                        @1150000
- Dao                              @1920000
- Tangerine                        @2463000
- SpuriousDragon                   @2675000
- Byzantium                        @4370000
- Constantinople                   @7280000
- Petersburg                       @7280000
- Istanbul                         @9069000
- MuirGlacier                      @9200000
- Berlin                           @12244000
- London                           @12965000
- ArrowGlacier                     @13773000
- GrayGlacier                      @15050000
Merge hard forks:
- Paris                            @58750000000000000000000 (network is known to be merged)
Post-merge hard forks (timestamp based):
- Shanghai                         @1681338455
- Cancun                           @1710338135
2024-04-13T08:29:45.943989Z DEBUG reth::cli: Spawning stages metrics listener task
2024-04-13T08:29:45.951670Z  WARN provider::static_file: Could not find block or tx number on a range request segment=Headers number=19645374
2024-04-13T08:29:45.955238Z ERROR reth::cli: shutting down due to error
2024-04-13T08:29:51.255518Z  INFO reth::cli: Starting reth version="0.2.0-beta.5 (54f75cd)"

[onbjerg]

cc @joshieDo looks static files related

[github-actions[bot]]

This issue is stale because it has been open for 21 days with no activity.

[joshieDo]

should be fixed when https://github.com/paradigmxyz/reth/pull/8143 gets merged