ton
ton copied to clipboard
ton-blockchain
A node synchronization error is reported
[ 3][t 1][2023-12-13 05:14:49.766596611][liteserver.cpp:609][!litequery] requesting state for block ((0,7000000000000000,40579290):D58B2574BFDF83E40A7A060AF70519E9D3E6646CE401797C7DA1F3CF6A039DE0:4DAA1CC19D80AC17796EEF7B49B65A202AD10C89721248A28208C25E2301C582) [ 3][t 1][2023-12-13 05:14:49.769098599][liteserver.cpp:79][!litequery] aborted liteserver query: [Error : 651 : block is not applied] [ 3][t 3][2023-12-13 05:14:49.769105452][liteserver.cpp:79][!litequery] aborted liteserver query: [Error : 651 : block is not applied] [ 3][t 6][2023-12-13 05:14:49.769097517][liteserver.cpp:79][!litequery] aborted liteserver query: [Error : 651 : block is not applied] [ 3][t 1][2023-12-13 05:14:49.776108391][liteserver.cpp:79][!litequery] aborted liteserver query: [Error : 651 : block is not applied] [ 2][t 5][2023-12-13 05:14:50.216932803][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:539.9ms] [ 2][t 6][2023-12-13 05:14:50.669812403][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:154.1ms] [ 2][t 5][2023-12-13 05:14:50.769880613][Timer.cpp:84][!waitstate][&duration > max_duration_] SLOW: [name:applyblocktostate][duration:115.8ms] [ 2][t 6][2023-12-13 05:14:51.137423608][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:storecell][duration:305.5ms] [ 2][t 6][2023-12-13 05:14:51.311048143][Timer.cpp:84][!waitstate][&duration > max_duration_] SLOW: [name:applyblocktostate][duration:140.5ms] [ 2][t 2][2023-12-13 05:14:51.347841319][Timer.cpp:84][!waitstate][&duration > max_duration_] SLOW: [name:applyblocktostate][duration:145.2ms] [ 2][t 5][2023-12-13 05:14:51.554519696][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:storecell][duration:208.8ms] [ 2][t 5][2023-12-13 05:14:51.778593584][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:storecell][duration:224.0ms] [ 2][t 5][2023-12-13 05:14:52.184106245][Timer.cpp:84][!waitstate][&duration > max_duration_] SLOW: [name:applyblocktostate][duration:100.9ms] [ 2][t 6][2023-12-13 05:14:52.250823540][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:296.2ms] [ 3][t 3][2023-12-13 05:14:52.253292348][state-serializer.cpp:197][!serializer] serializing masterchain state [ w=-1 s=9223372036854775808 seq=34698987 ] [ 2][t 6][2023-12-13 05:14:52.557666671][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:storecell][duration:141.4ms] [ 2][t 4][2023-12-13 05:14:53.087015048][Timer.cpp:84][!waitstate][&duration > max_duration_] SLOW: [name:applyblocktostate][duration:144.9ms] [ 2][t 7][2023-12-13 05:14:53.143993025][Timer.cpp:84][!waitstate][&duration > max_duration_] SLOW: [name:applyblocktostate][duration:225.6ms] [ 2][t 3][2023-12-13 05:14:53.168657354][Timer.cpp:84][!waitstate][&duration > max_duration_] SLOW: [name:applyblocktostate][duration:205.8ms] [ 2][t 2][2023-12-13 05:14:53.183111987][Timer.cpp:84][!waitstate][&duration > max_duration_] SLOW: [name:applyblocktostate][duration:258.3ms] [ 2][t 1][2023-12-13 05:14:53.214256109][Timer.cpp:84][!waitstate][&duration > max_duration_] SLOW: [name:applyblocktostate][duration:302.4ms] [ 2][t 6][2023-12-13 05:14:53.236727292][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:366.1ms] [ 2][t 6][2023-12-13 05:14:53.566850359][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:storecell][duration:117.9ms] [ 2][t 6][2023-12-13 05:14:53.881749443][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:storecell][duration:314.8ms] [ 2][t 4][2023-12-13 05:14:54.209024288][Timer.cpp:84][!waitstate][&duration > max_duration_] SLOW: [name:applyblocktostate][duration:120.8ms] Segmentation fault (core dumped)
Same coming for us
{“error”:“Block (-1, -9223372036854775808, 34714712) not found in DB”}
@blocknodehub seems like you are using slow disks. We recommend to use NVM SSD. @ianlapinskii where you this error? http-ton-api?
@EmelyanenkoK We face this error on our own node instances. We deployed them with TON API service
@blocknodehub seems like you are using slow disks. We recommend to use NVM SSD. @ianlapinskii where you this error? http-ton-api?
But I have observed disk usage and the performance is perfectly adequate for node usage, and the compiled packages I use available on github have this problem regardless of the failed v2023.06 or v2023.11 versions https://github.com/ton-blockchain/ton/releases/download/v2023.11/ton-linux-x86_64.zip
@blocknodehub well, logs explicitly mentioned that operations (most of them disk intensive) are slow. Note, that for now IOPS are more important that overall read/write speed measured in bytes/sec. @ianlapinskii Is your issue resolved? Most probably it was related to the node not synced and should gone once node will catch up with the network.
@blocknodehub well, logs explicitly mentioned that operations (most of them disk intensive) are slow. Note, that for now IOPS are more important that overall read/write speed measured in bytes/sec. @ianlapinskii Is your issue resolved? Most probably it was related to the node not synced and should gone once node will catch up with the network.
I use ssd disks, but the nodes are out of sync I used the snapshot data from December 13th to start the synchronization Also, I will stop using ssd disks and report errors: Segmentation fault (core dumped)
[ 3][t 4][2023-12-14 10:17:11.331012855][manager.cpp:1618][!manager] groups=0 seqno=34722671 our_seqno=34733768 [ 3][t 7][2023-12-14 10:17:11.331273689][manager.cpp:1650][!archive] failed to download archive slice: [Error : 651 : no nodes] [ 2][t 4][2023-12-14 10:17:11.858231250][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:403.2ms] [ 2][t 4][2023-12-14 10:17:13.202072527][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:1298.2ms] [ 3][t 2][2023-12-14 10:17:13.332678655][manager.cpp:1618][!manager] groups=0 seqno=34722671 our_seqno=34733768 [ 3][t 2][2023-12-14 10:17:13.332865546][manager.cpp:1650][!archive] failed to download archive slice: [Error : 651 : no nodes] [ 2][t 4][2023-12-14 10:17:13.756944456][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:548.5ms] [ 2][t 4][2023-12-14 10:17:14.374314091][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:444.2ms] [ 2][t 4][2023-12-14 10:17:15.128569139][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:664.0ms] [ 3][t 3][2023-12-14 10:17:15.334240470][manager.cpp:1618][!manager] groups=0 seqno=34722671 our_seqno=34733768 [ 3][t 3][2023-12-14 10:17:15.334417864][manager.cpp:1650][!archive] failed to download archive slice: [Error : 651 : no nodes] [ 2][t 4][2023-12-14 10:17:15.905802570][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:773.4ms] [ 2][t 4][2023-12-14 10:17:16.285156124][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:241.4ms] [ 2][t 4][2023-12-14 10:17:16.619698878][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:280.7ms] [ 2][t 7][2023-12-14 10:17:16.989370041][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:330.7ms] [ 3][t 2][2023-12-14 10:17:17.335788028][manager.cpp:1618][!manager] groups=0 seqno=34722671 our_seqno=34733768 [ 3][t 2][2023-12-14 10:17:17.336013209][manager.cpp:1650][!archive] failed to download archive slice: [Error : 651 : no nodes] [ 2][t 7][2023-12-14 10:17:17.543432865][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:547.5ms] [ 2][t 6][2023-12-14 10:17:19.020737333][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:1474.2ms] [ 3][t 6][2023-12-14 10:17:19.338286837][manager.cpp:1618][!manager] groups=0 seqno=34722671 our_seqno=34733768 [ 3][t 6][2023-12-14 10:17:19.338545306][manager.cpp:1650][!archive] failed to download archive slice: [Error : 651 : no nodes] [ 2][t 7][2023-12-14 10:17:19.542545150][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:435.6ms] [ 2][t 7][2023-12-14 10:17:19.952032134][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:407.9ms] [ 2][t 7][2023-12-14 10:17:20.526505547][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:462.9ms] [ 3][t 6][2023-12-14 10:17:21.340100204][manager.cpp:1618][!manager] groups=0 seqno=34722671 our_seqno=34733768 [ 3][t 6][2023-12-14 10:17:21.340295540][manager.cpp:1650][!archive] failed to download archive slice: [Error : 651 : no nodes] [ 2][t 7][2023-12-14 10:17:21.470400481][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:884.2ms] [ 2][t 7][2023-12-14 10:17:21.922488807][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:446.0ms] [ 2][t 7][2023-12-14 10:17:22.218227943][Timer.cpp:84][!celldbin][&duration > max_duration_] SLOW: [name:gccell][duration:234.4ms
@ianlapinskii Is your issue resolved? Most probably it was related to the node not synced and should gone once node will catch up with the network.
@EmelyanenkoK Yeah, that's the point. We still experience this error for some of new blocks, and having this error just makes it impossible to process several transactions. Could you please let us know if it has something to do with nano transaction mints?
@blocknodehub well, logs explicitly mentioned that operations (most of them disk intensive) are slow. Note, that for now IOPS are more important that overall read/write speed measured in bytes/sec. @ianlapinskii Is your issue resolved? Most probably it was related to the node not synced and should gone once node will catch up with the network.
What can I do to restore the node