[Bug] Failed to start the node - thread 'main' panicked at 'Failed to start the node: io error: unexpected end of file', src/main.rs:64:28

[skvoo]

🐛 Bug Report

Dec 29 08:00:57 Ubuntu-2004-focal-64-minimal snarkos[24090]: 2021-12-29T07:00:57.763382Z DEBUG Validating the ledger up to block 78038 (88%)
Dec 29 08:00:59 Ubuntu-2004-focal-64-minimal snarkos[24090]: 2021-12-29T07:00:59.740252Z DEBUG Validating the ledger up to block 80039 (91%)
Dec 29 08:01:01 Ubuntu-2004-focal-64-minimal snarkos[24090]: 2021-12-29T07:01:01.717031Z DEBUG Validating the ledger up to block 82040 (93%)
Dec 29 08:01:03 Ubuntu-2004-focal-64-minimal snarkos[24090]: 2021-12-29T07:01:03.698157Z DEBUG Validating the ledger up to block 84041 (95%)
Dec 29 08:01:05 Ubuntu-2004-focal-64-minimal snarkos[24090]: 2021-12-29T07:01:05.696591Z DEBUG Validating the ledger up to block 86042 (98%)
Dec 29 08:01:07 Ubuntu-2004-focal-64-minimal snarkos[24090]: 2021-12-29T07:01:07.402390Z DEBUG Validating the ledger up to block 87755 (100%)
Dec 29 08:01:07 Ubuntu-2004-focal-64-minimal snarkos[24090]: thread 'main' panicked at 'Failed to start the node: io error: unexpected end of file', src/main.rs:64:28
Dec 29 08:01:07 Ubuntu-2004-focal-64-minimal snarkos[24090]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Dec 29 08:01:07 Ubuntu-2004-focal-64-minimal systemd[1]: aleod-miner.service: Main process exited, code=exited, status=101/n/a
Dec 29 08:01:07 Ubuntu-2004-focal-64-minimal systemd[1]: aleod-miner.service: Failed with result 'exit-code'.
Dec 29 08:01:17 Ubuntu-2004-focal-64-minimal systemd[1]: aleod-miner.service: Scheduled restart job, restart counter is at 1.
Dec 29 08:01:17 Ubuntu-2004-focal-64-minimal systemd[1]: Stopped Aleo Miner Testnet2.

Expected Behavior

Expected to start a node

Your Environment

Aleo version 11/12 Ubuntu 20.xx

[ljedrz]

Could you provide a backtrace (run with RUST_BACKTRACE=1 environment variable)?

[martin-luo]

I actually have the same problem, here is the trace:

thread 'main' panicked at 'Failed to start the node: io error: unexpected end of file', src/main.rs:64:28
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
   2: core::result::unwrap_failed
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/result.rs:1616:5
   3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   4: tokio::park::thread::CachedParkThread::block_on
   5: tokio::runtime::thread_pool::ThreadPool::block_on
   6: tokio::runtime::Runtime::block_on
   7: snarkos::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Here is the configuration of the miner machine:

CPU: AMD Ryzen 5 3600 Hexa-Core
CPU-B: 19914
RAM: 64GB DDR4
Drives: 2 x 512 GB NVMe SSD

Please let me know if you need further information.

[ljedrz]

Hmm, the backtrace is sadly not too helpful here; I'm suspecting some database file corruption, but I'd need to inspect a ledger snapshot to be sure.

[skvoo]

Could you provide a backtrace (run with RUST_BACKTRACE=1 environment variable)?

Sorry, I just cleared the ledger. Seems it's a db issue after long miner running

[jamjahal]

I'm having the same issue. Hope this did not exclude me from the reward snapshot, but would love to get resolved

Can provide a ledger snapshot if helpful but will need some guidance on how.

Here is the traceback with RUST_TRACEBACK='full':

                                at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/back[0/1969
bunwind.rs:90:5                                                                                                                  
   1:     0x55f74b9a227c - std::backtrace_rs::backtrace::trace_unsynchronized::h849d83492cbc0d59                                 
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/m
d.rs:66:5                                                                                                                        
   2:     0x55f74b9a227c - std::sys_common::backtrace::_print_fmt::he3179d37290f23d3                                             
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:67:5   
   3:     0x55f74b9a227c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h140f6925cad14324  
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:46:22  
   4:     0x55f74b33a28c - core::fmt::write::h31b9cd1bedd7ea38                                                                   
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/fmt/mod.rs:1150:17            
   5:     0x55f74b99b6f5 - std::io::Write::write_fmt::h1fdf66f83f70913e                                                          
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/io/mod.rs:1667:15              
   6:     0x55f74b9a3c10 - std::sys_common::backtrace::_print::he7ac492cd19c3189                                                 
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:49:5   
   7:     0x55f74b9a3c10 - std::sys_common::backtrace::print::hba20f8920229d8e8                                                  
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:36:9   
   8:     0x55f74b9a3c10 - std::panicking::default_hook::{{closure}}::h714d63979ae18678                                          
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:210:50            
   9:     0x55f74b9a445a - std::panicking::default_hook::hf1afb64e69563ca8                                                       
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:227:9             
  10:     0x55f74b9a445a - std::panicking::rust_panic_with_hook::h02231a501e274a13                                               
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:624:17            
  11:     0x55f74b9a3d5a - std::panicking::begin_panic_handler::{{closure}}::h5b5e738edf361af7                                   
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:521:13            
  12:     0x55f74b9a2994 - std::sys_common::backtrace::__rust_end_short_backtrace::h601a115def7987b1                             
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:141:18 
  13:     0x55f74b9a3cdd - rust_begin_unwind                                                                                     
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5             
  14:     0x55f74a8b3511 - core::panicking::panic_fmt::h7a58c8fffc5559a4                                                         
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:101:14           
  15:     0x55f74a8b3603 - core::result::unwrap_failed::h1036b5c6eb8030dd                                                        
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/result.rs:1617:5              
  16:     0x55f74ad53fa5 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::heea899e082f4bb51 
  17:     0x55f74af19127 - tokio::park::thread::CachedParkThread::block_on::h5b47422b49880726                                    
  18:     0x55f74b01a04c - tokio::runtime::thread_pool::ThreadPool::block_on::ha9889acb991e1231                                  
  19:     0x55f74afaf6ae - tokio::runtime::Runtime::block_on::h76e3d3f8ee3f0127                                                  
  20:     0x55f74b00f671 - snarkos::main::h302b0487030eaa10                                                                      
  21:     0x55f74afaf243 - std::sys_common::backtrace::__rust_begin_short_backtrace::h998ddef554826ca1                           
  22:     0x55f74b09d8dd - std::rt::lang_start::{{closure}}::h98993ea766408035                                                   
  23:     0x55f74b9a4bbf - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h2790017aba790142
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:259:13        
  24:     0x55f74b9a4bbf - std::panicking::try::do_call::hd5d0fbb7d2d2d85d                                                       
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:403:40            
  25:     0x55f74b9a4bbf - std::panicking::try::h675520ee37b0fdf7                                                                
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:367:19            
  26:     0x55f74b9a4bbf - std::panic::catch_unwind::h803430ea0284ce79                                                           
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panic.rs:129:14                
  27:     0x55f74b9a4bbf - std::rt::lang_start_internal::{{closure}}::h3a398a8154de3106                                          
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/rt.rs:45:48                    
  28:     0x55f74b9a4bbf - std::panicking::try::do_call::hf60f106700df94b2                                                       
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:403:40            
  29:     0x55f74b9a4bbf - std::panicking::try::hb2022d2bc87a9867                                                                
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:367:19            
  30:     0x55f74b9a4bbf - std::panic::catch_unwind::hbf801c9d61f0c2fb                                                           
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panic.rs:129:14                
  31:     0x55f74b9a4bbf - std::rt::lang_start_internal::hdd488b91dc742b96                                                       
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/rt.rs:45:20                    
  32:     0x55f74b00f812 - main                                                                                                  
  33:     0x7f86a5ee80b3 - __libc_start_main                                                                                     
  34:     0x55f74a923d8e - _start                                                                                                
  35:                0x0 - <unknown>

git pull shows that i am up to date.

last pull at 2022-01-21 18:49:36.754111598 +0000

[jamjahal]

Hi i'm following up on this. @skvoo did clearing the ledger solve the issue?

How does one clear the ledger?

[ljedrz]

snarkos clean

[ljedrz]

Should be fixed in testnet3 thanks to https://github.com/AleoHQ/snarkOS/pull/1639, specifically with its https://github.com/AleoHQ/snarkOS/pull/1542 component.

[ljedrz]

Closing, as it doesn't seem to have triggered in a long time now.