mina
mina copied to clipboard
Published
20 hours ago •
MinaProtocol
MinaProtocol
UMT-upgrade-mechanism-testing Node is stuck in catchup
Preliminary Checks
- [X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
- [X] This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions
Description
My node has been stuck at block 27 for over an hour. Node has seen block 32, but is hanging out in catchup and just stacking up transactions. (This node is not a block producer, just running archive.)
Nothing really interesting in logs - will attach 2 sets of log snapshots - 1 taken before restart, and one ~1 hour after restart - still stuck in catchup.
I then stopped the node, deleted .mina-config, and restarted. Node came right up, in synch, but at block 1 with max observed height of 1. Took just over 11 minutes to see other blocks, then saw tip of 40, caught up to 40 / in synch after 17 mins from restart.
I should have preserved .mina-config directory, but neglected to do that.
Steps to Reproduce
- Start node, progressed normally in synch from blocks 1 - 27.
- Got stuck at block 27
- Restarted node
- Progressed to 28, but then stuck in catchup and never got into sync (waited 55 minutes)
- Stopped node
- Deleted .mina-config directory
- Started node, made it back to sync. ...
Expected Result
Node would not fall out of sync, and would be able to catchup without deleting .mina-config.
Actual Result
Node seems to have gotten stuck at block 27 and would not get back in synch until I deleted .mina-config.
Daemon version
Commit [DIRTY]2025a732df43c6735a576aff907252f74cf2ef35 on branch testing/hard-fork-internal
How frequently do you see this issue?
Rarely
What is the impact of this issue on your ability to run a node?
High
Status
**Status when issue noticed:**
---
Mina daemon status
-----------------------------------
Global number of accounts: 201806
Block height: 27
Max observed block height: 32
Max observed unvalidated block height: 32
Local uptime: 6h9m7s
Ledger Merkle root: jxv1qo8dQ6o3D41MHzPuuFtD5k9cjRd6gR1utt3CESsVbgPmSkL
Protocol state hash: 3NLmDdDQmG14HpsgY4fspTXswrHzSd1FHpucfZTsHcfgRTj8N5Tx
Chain id: 100ef4660dd2f1dc184e5730cc21f5ecfbe2f7b24af237fe049ca425cea6633e
Git SHA-1: [DIRTY]2025a732df43c6735a576aff907252f74cf2ef35
Configuration directory: /home/minar/.mina-config
Peers: 29
User_commands sent: 0
SNARK worker: None
SNARK work fee: 100000000
Sync status: Catchup
Catchup status:
Finished: 28
Block producers running: 0
Coinbase receiver: Block producer
Best tip consensus time: epoch=0, slot=338
Best tip global slot (across all hard-forks): 338
Consensus time now: epoch=0, slot=366
Consensus mechanism: proof_of_stake
Consensus configuration:
Delta: 0
k: 290
Slots per epoch: 7140
Slot duration: 3m
Epoch duration: 14d21h
Chain start timestamp: 2024-02-23 10:00:00.000000Z
Acceptable network delay: 3m
Addresses and ports:
External IP:
Bind IP: 0.0.0.0
Libp2p PeerID:
Libp2p port: 8304
Client port: 8301
Metrics:
block_production_delay: 7 (0 0 0 0 0 0 0)
transaction_pool_diff_received: 52
transaction_pool_diff_broadcasted: 0
transactions_added_to_pool: 1203
transaction_pool_size: 602
---
**Status ~1 hour after first restart:**
Mina daemon status
-----------------------------------
Global number of accounts: 201806
Block height: 28
Max observed block height: 39
Max observed unvalidated block height: 39
Local uptime: 55m43s
Ledger Merkle root: jxUp9fy4bWaqZFXeDUwzS56no51m4nuWRViP1hDkVEkqiMG3RxP
Protocol state hash: 3NLG9AgKgHzaQKWQKtdbuiEZxHpGV1MuKPSjgvXyFjQgrURs5FrX
Chain id: 100ef4660dd2f1dc184e5730cc21f5ecfbe2f7b24af237fe049ca425cea6633e
Git SHA-1: [DIRTY]2025a732df43c6735a576aff907252f74cf2ef35
Configuration directory: /home/minar/.mina-config
Peers: 29
User_commands sent: 0
SNARK worker: None
SNARK work fee: 100000000
Sync status: Catchup
Catchup status:
To build breadcrumb: 1
To initial validate: 0
Finished: 3
To download: 0
Waiting for parent to finish: 4
To verify: 0
Block producers running: 0
Coinbase receiver: Block producer
Best tip consensus time: epoch=0, slot=339
Best tip global slot (across all hard-forks): 339
Consensus time now: epoch=0, slot=387
Consensus mechanism: proof_of_stake
Consensus configuration:
Delta: 0
k: 290
Slots per epoch: 7140
Slot duration: 3m
Epoch duration: 14d21h
Chain start timestamp: 2024-02-23 10:00:00.000000Z
Acceptable network delay: 3m
Addresses and ports:
External IP:
Bind IP: 0.0.0.0
Libp2p PeerID:
Libp2p port: 8304
Client port: 8301
Metrics:
block_production_delay: 7 (0 0 0 0 0 0 0)
transaction_pool_diff_received: 78
transaction_pool_diff_broadcasted: 0
transactions_added_to_pool: 0
transaction_pool_size: 0
---
**Status ~18mins after delete .mina-config and restart:**
mina client status
Mina daemon status
-----------------------------------
Global number of accounts: 201806
Block height: 40
Max observed block height: 40
Max observed unvalidated block height: 40
Local uptime: 17m58s
Ledger Merkle root: jwP8ayfGmq2sHrrurW2d78WY9YREVN1ByDbkGkuoQNi8NVSKbiR
Protocol state hash: 3NLm2Z81fG2LG5zMTMy2aYZVYrkr8st6EnLnFL3SWTJM2fRTDqE8
Chain id: 100ef4660dd2f1dc184e5730cc21f5ecfbe2f7b24af237fe049ca425cea6633e
Git SHA-1: [DIRTY]2025a732df43c6735a576aff907252f74cf2ef35
Configuration directory: /home/minar/.mina-config
Peers: 27
User_commands sent: 0
SNARK worker: None
SNARK work fee: 100000000
Sync status: Synced
Catchup status:
To build breadcrumb: 0
To initial validate: 0
Finished: 41
To download: 0
Waiting for parent to finish: 0
To verify: 0
Block producers running: 0
Coinbase receiver: Block producer
Best tip consensus time: epoch=0, slot=391
Best tip global slot (across all hard-forks): 391
Consensus time now: epoch=0, slot=393
Consensus mechanism: proof_of_stake
Consensus configuration:
Delta: 0
k: 290
Slots per epoch: 7140
Slot duration: 3m
Epoch duration: 14d21h
Chain start timestamp: 2024-02-23 10:00:00.000000Z
Acceptable network delay: 3m
Addresses and ports:
External IP:
Bind IP: 0.0.0.0
Libp2p PeerID:
Libp2p port: 8304
Client port: 8301
Metrics:
block_production_delay: 7 (0 0 0 0 0 0 0)
transaction_pool_diff_received: 10
transaction_pool_diff_broadcasted: 0
transactions_added_to_pool: 0
transaction_pool_size: 0
Additional information
I exported logs at each of the 3 intervals reported above. I'm only sharing the last export because it should capture all three intervals.
