ironfish icon indicating copy to clipboard operation
ironfish copied to clipboard

ironfish node stop syncing on [email protected]

Open darrencavell opened this issue 2 years ago • 4 comments

What happened?

node keeps getting disconnected and the current node is idle

Connection closing WebSocket for Uwmwtea (ynHwCi): read ECONNRESET                                                                  
PEER Uwmwtea (ynHwCi) STATE CONNECTING -> DISCONNECTED                                                                                        
CONN WebSocket unidentified STATE CONNECTING -> DISCONNECTED                                                                                  
Connection closing WebSocket for NhkhRJb (ironbarra43): read ECONNRESET 
PEER NhkhRJb (ironbarra43) STATE CONNECTING -> DISCONNECTED
RECV WebSocket SPuAELH: PeerListRequest (10)
RECV WebRtc f+dl8ZG: PeerListRequest (10)                                                                                                     
CONN WebSocket unidentified STATE CONNECTING -> WAITING_FOR_IDENTITY
SEND WebSocket unidentified: Identify (6)                                                                                                     
Received non-buffer message {"type":0,"destinationIdentity":null,"disconnectUntil":1665048864765,"reason":1,"sourceIdentity":"1FOWDS/vsKU+Yk6b
6zcCHq5+8pCAZ+xzWfH0pfj9qUY="}                                                                                                                
CONN WebSocket unidentified STATE WAITING_FOR_IDENTITY -> DISCONNECTED                                                                        
Connection closing WebSocket for 1FOWDS/ (ahamohar): Received non-buffer message
PEER 1FOWDS/ (ahamohar) STATE CONNECTING -> DISCONNECTED          
CONN WebRtc unidentified STATE DISCONNECTED -> CONNECTING
PEER 6WuINBQ (aabo43) STATE DISCONNECTED -> CONNECTING                                                                                        
Attempted to establish a WebRTC connection to JVTIBa+ (Bula_va_Ironfish), but couldn't find a peer to broker the connection.
CONN WebSocket unidentified STATE DISCONNECTED -> CONNECTING                                                                                  
PEER JVTIBa+ (Bula_va_Ironfish) STATE DISCONNECTED -> CONNECTING
CONN WebRtc unidentified STATE DISCONNECTED -> CONNECTING                                                                                     
PEER EmzfRcM (slafi) STATE DISCONNECTED -> CONNECTING
RECV WebRtc HnFsnuN: PeerListRequest (10)                                                                                                     
RECV WebRtc Ud8uLUs: Disconnecting (0)                                                                                                        
2fdadA+ (rkds75) requested we disconnect until 1665048865000. Current time is 1665048565693
RECV WebRtc Ud8uLUs: Disconnecting (0)                                                                                                        
2fdadA+ (rkds75) requested we disconnect until 1665048865000. Current time is 1665048565693
RECV WebRtc Ud8uLUs: Disconnecting (0)                                                                                                        
SEND WebRtc Ud8uLUs: Disconnecting (0)                      
2fdadA+ (rkds75) requested we disconnect until 1665048865000. Current time is 1665048565693
Ignoring signaling request from 2fdadA+hprh/3mvgWuewyBD/htyDGUsPGDU4rRWY5XI=, at max peers
SEND WebRtc Ud8uLUs: Disconnecting (0)                                                                                                        
Ignoring signaling request from 2fdadA+hprh/3mvgWuewyBD/htyDGUsPGDU4rRWY5XI=, at max peers
SEND WebRtc Ud8uLUs: Disconnecting (0)                                                                                                        
Ignoring signaling request from 2fdadA+hprh/3mvgWuewyBD/htyDGUsPGDU4rRWY5XI=, at max peers
Ignoring signaling request from 2fdadA+hprh/3mvgWuewyBD/htyDGUsPGDU4rRWY5XI=, at max peers
Version              0.1.49 @ ec57ed3
Node                 STARTED
Node Name            DarrzDC
Block Graffiti       DarrzDC
Memory               Heap: 109.72 MiB -> 126.67 MiB / 2.01 GiB (5.3%), RSS: 1.56 GiB (20.8%), Free: 4.87 GiB (35.2%)
CPU                  Cores: 16, Current: 67.6%
P2P Network          CONNECTED - In: 13.81 KB/s, Out: 2.79 KB/s, peers 44
Mining               STARTED - 0 miners, 0 mined
Mem Pool             Size: 0 tx, Bytes: 0 B
Syncer               IDLE - avg time to add block 13801.5 ms, progress: 97.82%
Blockchain           0000000000003f258accc4de083e8b4a45491676ea33354a1a06e01e09557f1f (210966), Since HEAD: 78h (NOT SYNCED)
Accounts             0000000000003f258accc4de083e8b4a45491676ea33354a1a06e01e09557f1f (210966)
Telemetry            STARTED - 6620 <- 0 pending
Workers              STARTED - 0 -> 0 / 6 - 0 jobs Δ, 0 jobs/s

Version

0.1.49

Debug output

Iron Fish version               0.1.49 @ ec57ed3
Iron Fish library               0.0.26 @ ec57ed3
Operating system                Linux x64
CPU model(s)                    11th Gen Intel(R) Core(TM) i7-11700 @ 2.50GHz
CPU threads                     16
RAM total                       7.53 GiB
Heap total                      2.01 GiB
Node version                    v16.17.1
ironfish in PATH                true
Garbage Collector Exposed       false
Telemetry enabled               true
Node name                       DarrzDC
Block graffiti                  DarrzDC

Relevant log output

No response

Graffiti

DarrzDC

darrencavell avatar Oct 06 '22 09:10 darrencavell

it keeps getting lower sync percentage and goes idle again

Version              0.1.49 @ ec57ed3
Node                 STARTED
Node Name            DarrzDC
Block Graffiti       DarrzDC
Memory               Heap: 176.05 MiB -> 221.39 MiB / 2.01 GiB (8.5%), RSS: 1.42 GiB (18.9%), Free: 4.38 GiB (41.7%)
CPU                  Cores: 16, Current: 92.8%
P2P Network          CONNECTED - In: 23 B/s, Out: 1.07 KB/s, peers 48
Mining               STARTED - 0 miners, 0 mined
Mem Pool             Size: 0 tx, Bytes: 0 B
Syncer               IDLE - avg time to add block 23531.57 ms, progress: 97.74%
Blockchain           0000000000001b02ed8a33d850bd99be622584c4d7ebf9ff21b414948dd2ba3d (211684), Since HEAD: 81h 8m (NOT SYNCED)
Accounts             0000000000001b02ed8a33d850bd99be622584c4d7ebf9ff21b414948dd2ba3d (211684)
Telemetry            STARTED - 33157 <- 0 pending
Workers              STARTED - 0 -> 1 / 6 - 0 jobs Δ, 0 jobs/s

darrencavell avatar Oct 07 '22 00:10 darrencavell

I also have pending transactions... And my balance becomes zero:

 Status     Creator Hash                               Miner Fee Fee ($IRON)         Notes Spends Expiration Sequence 
 ────────── ─────── ────────────────────────────────── ───────── ─────────────────── ───── ────── ─────────────────── 
 pending    x       17407014f5557e90baa1d197c327b9e2e… x         0.00000274          275   3      209180              
 pending    x       28f35f8c7e90bf876108ee87aa13ee872… x         0.00000280          281   3      211059              
 pending    x       376e77e51afa303d3af0dd2c8c35b3beb… x         0.00000305          306   2      192368              
 pending    x       43f9ec5543b656576e03d31d6dc9c9223… x         0.00000273          274   3      208701              
 pending    x       4a07f5486c4605cb7572afc5399c42a27… x         0.00000304          305   2      197525              
 pending    x       5a4423f46f9103ff891823f25a351efeb… x         0.00000319          320   2      197077              
 pending    x       6d81415c3720e50272d49d3592a6da139… x         0.00000272          273   3      209579              
 pending    x       89aa56c095e5ff2d775bd355697c22830… x         0.00000271          272   3      209795              
 pending    x       9a4e1093eb2d7aa6a460fba72f3f1aa1d… x         0.00000286          287   1      204971              
 pending    x       b510121ad159f83e42c53f983a3e998bc… x         0.00000333          334   1      195463              
 pending    x       b64cac500d01e782532662bc0e5092ea7… x         0.00000272          273   3      208930              
 pending    x       e8588d40ae8a90be9be2993dd1fb6126e… x         0.00000279          280   3      210300              
 pending    x       e8c493cbb05dd48f7aa4ce32d0d2d29f4… x         0.00000267          268   3      209344              
 pending    x       feb68c4f4c7d9d418a4e8dd795134893c… x         0.00000273          274   3      210041              
Account: mib383 (2212859)
Balance: $IRON 0.00000000
Done in 8.74s.

mib383 avatar Oct 07 '22 18:10 mib383

i try to restart if it goes idle for some amount of time, here's another idle after restarting the node, and the progress doesn't seem to increase

Version              0.1.49 @ ec57ed3
Node                 STARTED
Node Name            DarrzDC
Block Graffiti       DarrzDC
Memory               Heap: 239.91 MiB -> 327.14 MiB / 2.01 GiB (11.6%), RSS: 2.43 GiB (32.3%), Free: 3.37 GiB (55.2%)
CPU                  Cores: 16, Current: 96.4%
P2P Network          CONNECTED - In: 28 B/s, Out: 684 B/s, peers 49
Mining               STARTED - 0 miners, 0 mined
Mem Pool             Size: 0 tx, Bytes: 0 B
Syncer               IDLE - avg time to add block 41530.38 ms, progress: 97.40%
Blockchain           000000000000284dde82181fb7540b724bfda593d49a9a7d07158d448f176a69 (212819), Since HEAD: 94h 13m (NOT SYNCED)
Accounts             000000000000284dde82181fb7540b724bfda593d49a9a7d07158d448f176a69 (212819)
Telemetry            STARTED - 167446 <- 0 pending
Workers              STARTED - 0 -> 0 / 6 - 0 jobs Δ, 0 jobs/s

darrencavell avatar Oct 08 '22 09:10 darrencavell

I have exactly the same. Since 0.1.45

Node                 STARTED
Node Name            DiElektra
Block Graffiti       DiElektra
Memory               Heap: 50.11 MiB -> 83.17 MiB / 4.01 GiB (1.2%), RSS: 1.32 GiB (8.5%), Free: 13.75 GiB (12.0%)   
CPU                  Cores: 6, Current: 109.5%
P2P Network          CONNECTED - In: 47.98 KB/s, Out: 1.85 MB/s, peers 32 
Mining               STARTED - 0 miners, 0 mined
Mem Pool             Size: 0 tx, Bytes: 0 B
Syncer               SYNCING - 0.06 blocks per seconds, avg time to add block 6815.78 ms, progress: 97.48%
Blockchain           00000000000006729b153d81ccecf10352fe41ca1fa8d8a877b8b14b1107920d (223709), Since HEAD: 96h 1m (NOT SYNCED) 
Accounts             00000000000535a54873db44ac968c66673b760461196c0e5edfafb0313f4e47 (134423)
Telemetry            STARTED - 0 <- 1 pending
Workers              STARTED - 0 -> 1 / 5 - 0 jobs Δ, 280.12 jobs/s    ```

D-Elektra avatar Oct 16 '22 01:10 D-Elektra

Hey @darrencavell and @D-Elektra thanks for reporting an issue and sorry for the late reply. We've made some measurable improvements on syncing speed which should be seen either in v0.1.50 or the next version. So it should be a lot less likely for the node to go out of sync

  • https://github.com/iron-fish/ironfish/pull/2437
  • https://github.com/iron-fish/ironfish/pull/2456
  • https://github.com/iron-fish/ironfish/pull/2464
  • https://github.com/iron-fish/ironfish/pull/2491

That being said, finding peers to sync from is still an issue sometimes which would cause the syncer to go IDLE. We're working on this in some of our upcoming projects on our roadmap so hopefully that should be addressed. I will leave this issue open because we have ongoing work but expect a while before another update as this is a longer running effort

danield9tqh avatar Nov 02 '22 18:11 danield9tqh

Thank you, guys! With version 0.1.50 I'm finally synced for the first time in 1.5 months ❤️

D-Elektra avatar Nov 03 '22 22:11 D-Elektra