Not synchronized after a certain block

[smithy9901]

log

[cardano-sl.*production*:Info:ThreadId 364] [2020-03-20 09:11:54.51 UTC] New wallet API has STARTED!
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:11:54.51 UTC] createAndArchiveCheckpoints is starting...
[cardano-sl.diffusion:Notice:ThreadId 338] [2020-03-20 09:11:54.51 UTC] subscriptionWorker: subscribing to NodeId 52.68.216.149:3000:0
[cardano-sl.*production*:Info:ThreadId 356] [2020-03-20 09:11:58.51 UTC] Triggering recovery because we need it
[cardano-sl.*production*:Info:ThreadId 354] [2020-03-20 09:11:59.88 UTC] streaming start
[cardano-sl.*production*:Info:ThreadId 398] [2020-03-20 09:12:01.65 UTC] Trying to apply blocks w/o rollback. First 3: [MainBlockHeader:
    hash: c7a98cd7c70967ab3e3f72e0d3a09571154b4aa7b99b179a7de68dae7bf66d8c
    previous block: 8c9eba581588802754a3eb86f2eb52d65b3ea4f4cf792243e34ec47640916c9f
    slot: 0th slot of 176th epoch
    difficulty: 3799518
    leader: pub:26566e86
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:26566e86, dPk = pub:f14f712d } }
    block: v1.0.0
    software: cardano-sl:2
, MainBlockHeader:
    hash: 93c2281de990021c69b16088d27823cb2d511785e8f6978cffac277eb834c21e
    previous block: c7a98cd7c70967ab3e3f72e0d3a09571154b4aa7b99b179a7de68dae7bf66d8c
    slot: 1st slot of 176th epoch
    difficulty: 3799519
    leader: pub:1bc97a2f
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:1bc97a2f, dPk = pub:61261a95 } }
    block: v1.0.0
    software: cardano-sl:2
, MainBlockHeader:
    hash: 592bb8b6b15bfa13b6657805add844c64408841a452c8e1eb4f78c959352e387
    previous block: 93c2281de990021c69b16088d27823cb2d511785e8f6978cffac277eb834c21e
    slot: 2nd slot of 176th epoch
    difficulty: 3799520
    leader: pub:50733161
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:50733161, dPk = pub:e8c03a03 } }
    block: v1.0.0
    software: cardano-sl:2
]
Last 3: [MainBlockHeader:
    hash: 5b392c3af0b06cac6b5681b2904ca1e16f925d69f61e09e220adad35db7b88e0
    previous block: 86ff31ff657d821b47508ce912720591bac2f1f2a94ee7ebf864c1d355a360c8
    slot: 61st slot of 176th epoch
    difficulty: 3799579
    leader: pub:9a6fa343
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:9a6fa343, dPk = pub:8b532076 } }
    block: v1.0.0
    software: cardano-sl:2
, MainBlockHeader:
    hash: 9a1fcae46aa94259d80f02aeb056cb75c5a828853b0e2174965dc93371c393b8
    previous block: 5b392c3af0b06cac6b5681b2904ca1e16f925d69f61e09e220adad35db7b88e0
    slot: 62nd slot of 176th epoch
    difficulty: 3799580
    leader: pub:0bdb1f5e
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:0bdb1f5e, dPk = pub:5fddeeda } }
    block: v1.0.0
    software: cardano-sl:2
, MainBlockHeader:
    hash: 1062bd9adc915a08429535c95f4819c0cb703feaff88e7506cb86fbd4edb8fbc
    previous block: 9a1fcae46aa94259d80f02aeb056cb75c5a828853b0e2174965dc93371c393b8
    slot: 63rd slot of 176th epoch
    difficulty: 3799581
    leader: pub:26566e86
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:26566e86, dPk = pub:f14f712d } }
    block: v1.0.0
    software: cardano-sl:2
]
[cardano-sl.*production*:Info:ThreadId 398] [2020-03-20 09:12:01.65 UTC] Verifying and applying blocks...
[cardano-sl.*production*:Info:ThreadId 398] [2020-03-20 09:12:01.65 UTC] slogVerifyBlocks: Consensus era is Original
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:13:00.79 UTC] pruneAndCompress pruned 1 old archives.
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:13:00.87 UTC] pruneAndCompress compressed 1 archives.

^C
root@proxy07 /opt/cardano # tail -f /opt/cardano/storage/state-wallet-mainnet/logs/node
    difficulty: 3799581
    leader: pub:26566e86
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:26566e86, dPk = pub:f14f712d } }
    block: v1.0.0
    software: cardano-sl:2
]
[cardano-sl.*production*:Info:ThreadId 398] [2020-03-20 09:12:01.65 UTC] Verifying and applying blocks...
[cardano-sl.*production*:Info:ThreadId 398] [2020-03-20 09:12:01.65 UTC] slogVerifyBlocks: Consensus era is Original
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:13:00.79 UTC] pruneAndCompress pruned 1 old archives.
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:13:00.87 UTC] pruneAndCompress compressed 1 archives.
[cardano-sl.*production*:Info:ThreadId 356] [2020-03-20 09:14:50.03 UTC] Checking if we need recovery as a safety measure
[cardano-sl.*production*:Info:ThreadId 356] [2020-03-20 09:15:20.04 UTC] Checking if we need recovery as a safety measure
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:18:00.88 UTC] createAndArchiveCheckpoints is starting...
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:19:06.80 UTC] pruneAndCompress pruned 1 old archives.
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:19:06.88 UTC] pruneAndCompress compressed 1 archives.
[cardano-sl.*production*:Info:ThreadId 356] [2020-03-20 09:19:18.32 UTC] Checking if we need recovery as a safety measure
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:43:32.90 UTC] pruneAndCompress pruned 1 old archives.
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:43:32.98 UTC] pruneAndCompress compressed 1 archives.
[cardano-sl.node:Warning:ThreadId 817] [2020-03-20 09:45:36.58 UTC] Failed to add headers from "52.68.216.149:3000:0" to block retrieval queue: queue is full
[cardano-sl.node:Warning:ThreadId 821] [2020-03-20 09:45:54.68 UTC] Failed to add headers from "52.68.216.149:3000:0" to block retrieval queue: queue is full
[cardano-sl.node:Warning:ThreadId 825] [2020-03-20 09:46:15.47 UTC] Failed to add headers from "52.68.216.149:3000:0" to block retrieval queue: queue is full
[cardano-sl.node:Warning:ThreadId 829] [2020-03-20 09:46:36.44 UTC] Failed to add headers from "52.68.216.149:3000:0" to block retrieval queue: queue is full
[cardano-sl.node:Warning:ThreadId 833] [2020-03-20 09:46:54.66 UTC] Failed to add headers from "52.68.216.149:3000:0" to block retrieval queue: queue is full
[cardano-sl.node:Warning:ThreadId 837] [2020-03-20 09:47:15.68 UTC] Failed to add headers from "52.68.216.149:3000:0" to block retrieval queue: queue is full
[cardano-sl.node:Warning:ThreadId 841] [2020-03-20 09:47:36.06 UTC] Failed to add headers from "52.68.216.149:3000:0" to block retrieval queue: queue is full
[cardano-sl.node:Warning:ThreadId 845] [2020-03-20 09:47:56.89 UTC] Failed to add headers from "52.68.216.149:3000:0" to block retrieval queue: queue is full
[cardano-sl.node:Warning:ThreadId 849] [2020-03-20 09:48:14.69 UTC] Failed to add headers from "52.68.216.149:3000:0" to block retrieval queue: queue is full
[cardano-sl.*production*:Info:ThreadId 366] [2020-03-20 09:48:32.99 UTC] createAndArchiveCheckpoints is starting...

[erikd]

There is a Control-C in that output, Why?

What is the evidience that its failing to sync?

What version are you running? Git tag? Git hash?

[erikd]

Please do not open new issues related to older ones without closing the older issue.

Closing this in favor of #4282

[cyberbudy]

The same problem here. I'm using the latest version 3.2.0

[diffusion:NOTICE:ThreadId 294] [2020-07-17 14:15:13.68 UTC] subscriptionWorker: subscribing to NodeId 13.114.110.189:3000:0
[node:INFO:ThreadId 332] [2020-07-17 14:15:16.73 UTC] Triggering recovery because we need it
[diffusion:NOTICE:ThreadId 294] [2020-07-17 14:15:28.68 UTC] subscriptionWorker: lost connection to NodeId 13.114.110.189:3000:0 Exceptional (TransportError ConnectTimeout "Timed out")
[diffusion.outboundqueue.self:WARNING:ThreadId 411] [2020-07-17 14:15:28.68 UTC] sending MsgRequestBlockHeaders Nothing to NodeId 13.114.110.189:3000:0 failed with TransportError ConnectTimeout "Timed out" :: SomeException
[node:ERROR:ThreadId 332] [2020-07-17 14:15:28.68 UTC] recoveryTriggerWorkerTransportError ConnectTimeout "Timed out"
[node:INFO:ThreadId 332] [2020-07-17 14:15:44.69 UTC] Triggering recovery because we need it
[diffusion:NOTICE:ThreadId 294] [2020-07-17 14:15:48.68 UTC] subscriptionWorker: subscribing to NodeId 13.229.193.102:3000:0
[diffusion:NOTICE:ThreadId 294] [2020-07-17 14:16:03.69 UTC] subscriptionWorker: lost connection to NodeId 13.229.193.102:3000:0 Exceptional (TransportError ConnectTimeout "Timed out")
[node:INFO:ThreadId 332] [2020-07-17 14:16:05.69 UTC] Triggering recovery because we need it

[smithy9901]

version 3.2.0 builded yesterday from the "develop" branch.

[vhendriks81]

I believe you should be upgrading to cardano-node + cardano-wallet etc as cardano-sl is not being supported anymore. This is all related to the shelley hard fork which is happening "any time now"