Acala
Acala copied to clipboard
Published
20 hours ago •
AcalaNetwork
AcalaNetwork
Acala mainnet stuck on finalized block #5446885
Describe the bug A new acala mainnet node, which has been syncing from genesis, is stuck on finalized block #5446885
Expected Behavior Sync from genesis till the latest block without issues.
Current Behavior Suck on finalized block #5446885.
Steps to Reproduce
- Start acala client without database
- Wait a few days
Additional context
- Acala version: acala 2.25.0-eb961734875
- OS: Ubuntu 22.04.3 LTS
- Command line options:
/usr/local/bin/acala \
--base-path=/var/lib/acala/data \
--chain=acala \
--name=xLabs \
# Pruning Acala DB
# Note: paritydb database is recommend when pruning is enabled
--database=paritydb \
--state-pruning=200000 \
--blocks-pruning=200000 \
--rpc-cors=all \
--rpc-external \
--rpc-port=9933 \
--rpc-max-connections=500 \
--port=30333 \
--prometheus-external \
--prometheus-port=9615 \
-- \
# Pruning Relaychain DB
--database=paritydb \
--state-pruning=200000 \
--blocks-pruning=200000
- Logs:
Jun 19 13:35:30 m-acala-02 acala[3326290]: 2024-06-19 13:35:30 ✨ Imported #6417394 (0xc2f3…180b)
Jun 19 13:35:35 m-acala-02 acala[3326290]: 2024-06-19 13:35:35 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4339712 (0xe134…12da), finalized #4339712 (0xe134…12da), ⬇ 6.7kiB/s ⬆ 8.4kiB/s
Jun 19 13:35:35 m-acala-02 acala[3326290]: 2024-06-19 13:35:35 💤 Idle (9 peers), best: #6417394 (0xc2f3…180b), finalized #5446885 (0x1f31…4d80), ⬇ 21.7kiB/s ⬆ 7.6kiB/s
Jun 19 13:35:40 m-acala-02 acala[3326290]: 2024-06-19 13:35:40 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4339712 (0xe134…12da), finalized #4339712 (0xe134…12da), ⬇ 5.0kiB/s ⬆ 8.4kiB/s
Jun 19 13:35:40 m-acala-02 acala[3326290]: 2024-06-19 13:35:40 💤 Idle (9 peers), best: #6417394 (0xc2f3…180b), finalized #5446885 (0x1f31…4d80), ⬇ 38 B/s ⬆ 38 B/s
Jun 19 13:35:42 m-acala-02 acala[3326290]: 2024-06-19 13:35:42 ✨ Imported #6417395 (0x0040…92bf)
Jun 19 13:35:42 m-acala-02 acala[3326290]: 2024-06-19 13:35:42 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 13:35:42 m-acala-02 acala[3326290]: 2024-06-19 13:35:42 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -1024 to -2147483648. Reason: same small block request multiple times. Banned, disconnecting.
Jun 19 13:35:45 m-acala-02 acala[3326290]: 2024-06-19 13:35:45 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4339712 (0xe134…12da), finalized #4339712 (0xe134…12da), ⬇ 5.0kiB/s ⬆ 7.4kiB/s
Jun 19 13:35:45 m-acala-02 acala[3326290]: 2024-06-19 13:35:45 💤 Idle (9 peers), best: #6417395 (0x0040…92bf), finalized #5446885 (0x1f31…4d80), ⬇ 7.5kiB/s ⬆ 254.7kiB/s
Jun 19 13:35:50 m-acala-02 acala[3326290]: 2024-06-19 13:35:50 [Relaychain] ⚙️ Syncing 153.6 bps, target=#21283499 (40 peers), best: #4340480 (0x736b…c762), finalized #4340224 (0x6def…3e37), ⬇ 81.4kiB/s ⬆ 5.8kiB/s
Jun 19 13:35:50 m-acala-02 acala[3326290]: 2024-06-19 13:35:50 💤 Idle (9 peers), best: #6417395 (0x0040…92bf), finalized #5446885 (0x1f31…4d80), ⬇ 64 B/s ⬆ 64 B/s
Jun 19 13:35:55 m-acala-02 acala[3326290]: 2024-06-19 13:35:55 [Relaychain] ⚙️ Syncing 395.4 bps, target=#21283499 (40 peers), best: #4342457 (0xb309…349b), finalized #4342273 (0x3dd4…c76b), ⬇ 669.1kiB/s ⬆ 5.1kiB/s
Jun 19 13:35:55 m-acala-02 acala[3326290]: 2024-06-19 13:35:55 💤 Idle (9 peers), best: #6417395 (0x0040…92bf), finalized #5446885 (0x1f31…4d80), ⬇ 0.2kiB/s ⬆ 57 B/s
Jun 19 13:35:55 m-acala-02 acala[3326290]: 2024-06-19 13:35:55 ✨ Imported #6417396 (0x046b…375e)
Jun 19 13:36:00 m-acala-02 acala[3326290]: 2024-06-19 13:36:00 [Relaychain] ⚙️ Syncing 444.0 bps, target=#21283499 (40 peers), best: #4344677 (0x755b…f84d), finalized #4344320 (0x35fa…353c), ⬇ 241.2kiB/s ⬆ 2.3kiB/s
Jun 19 13:36:00 m-acala-02 acala[3326290]: 2024-06-19 13:36:00 💤 Idle (9 peers), best: #6417396 (0x046b…375e), finalized #5446885 (0x1f31…4d80), ⬇ 11.0kiB/s ⬆ 21.9kiB/s
Jun 19 13:36:05 m-acala-02 acala[3326290]: 2024-06-19 13:36:05 [Relaychain] ⚙️ Syncing 375.6 bps, target=#21283499 (40 peers), best: #4346555 (0xe369…a917), finalized #4346368 (0x5f87…c70a), ⬇ 189.9kiB/s ⬆ 3.3kiB/s
Jun 19 13:36:05 m-acala-02 acala[3326290]: 2024-06-19 13:36:05 💤 Idle (9 peers), best: #6417396 (0x046b…375e), finalized #5446885 (0x1f31…4d80), ⬇ 64 B/s ⬆ 64 B/s
Jun 19 13:36:06 m-acala-02 acala[3326290]: 2024-06-19 13:36:06 ✨ Imported #6417397 (0xe5cd…04c4)
Jun 19 13:36:10 m-acala-02 acala[3326290]: 2024-06-19 13:36:10 [Relaychain] ⚙️ Syncing 474.6 bps, target=#21283499 (40 peers), best: #4348928 (0xf168…c5a4), finalized #4348928 (0xf168…c5a4), ⬇ 225.4kiB/s ⬆ 2.1kiB/s
Jun 19 13:36:10 m-acala-02 acala[3326290]: 2024-06-19 13:36:10 💤 Idle (9 peers), best: #6417397 (0xe5cd…04c4), finalized #5446885 (0x1f31…4d80), ⬇ 18.2kiB/s ⬆ 7.5kiB/s
Jun 19 13:36:15 m-acala-02 acala[3326290]: 2024-06-19 13:36:15 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4348928 (0xf168…c5a4), finalized #4348928 (0xf168…c5a4), ⬇ 2.9kiB/s ⬆ 0.9kiB/s
Jun 19 13:36:15 m-acala-02 acala[3326290]: 2024-06-19 13:36:15 💤 Idle (9 peers), best: #6417397 (0xe5cd…04c4), finalized #5446885 (0x1f31…4d80), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
Jun 19 13:36:18 m-acala-02 acala[3326290]: 2024-06-19 13:36:18 ✨ Imported #6417398 (0x58ca…7c9a)
Jun 19 13:36:20 m-acala-02 acala[3326290]: 2024-06-19 13:36:20 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4348928 (0xf168…c5a4), finalized #4348928 (0xf168…c5a4), ⬇ 2.2kiB/s ⬆ 1.0kiB/s
Jun 19 13:36:20 m-acala-02 acala[3326290]: 2024-06-19 13:36:20 💤 Idle (9 peers), best: #6417398 (0x58ca…7c9a), finalized #5446885 (0x1f31…4d80), ⬇ 4.0kiB/s ⬆ 25.4kiB/s
Jun 19 13:36:25 m-acala-02 acala[3326290]: 2024-06-19 13:36:25 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4348928 (0xf168…c5a4), finalized #4348928 (0xf168…c5a4), ⬇ 3.1kiB/s ⬆ 1.2kiB/s
Jun 19 13:36:25 m-acala-02 acala[3326290]: 2024-06-19 13:36:25 💤 Idle (9 peers), best: #6417398 (0x58ca…7c9a), finalized #5446885 (0x1f31…4d80), ⬇ 44 B/s ⬆ 44 B/s
Jun 19 13:36:27 m-acala-02 acala[3326290]: 2024-06-19 13:36:27 [Relaychain] Sending fatal alert BadCertificate
Jun 19 13:36:30 m-acala-02 acala[3326290]: 2024-06-19 13:36:30 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4348928 (0xf168…c5a4), finalized #4348928 (0xf168…c5a4), ⬇ 138.3kiB/s ⬆ 77.0kiB/s
Jun 19 13:36:30 m-acala-02 acala[3326290]: 2024-06-19 13:36:30 💤 Idle (9 peers), best: #6417398 (0x58ca…7c9a), finalized #5446885 (0x1f31…4d80), ⬇ 1.5kiB/s ⬆ 0.8kiB/s
Jun 19 13:36:30 m-acala-02 acala[3326290]: 2024-06-19 13:36:30 ✨ Imported #6417399 (0x6d82…ec4b)
Jun 19 13:36:35 m-acala-02 acala[3326290]: 2024-06-19 13:36:35 [Relaychain] ⚙️ Syncing 469.0 bps, target=#21283499 (40 peers), best: #4351273 (0xf916…edea), finalized #4350976 (0x723c…71e0), ⬇ 224.3kiB/s ⬆ 10.5kiB/s
Jun 19 13:36:35 m-acala-02 acala[3326290]: 2024-06-19 13:36:35 💤 Idle (9 peers), best: #6417399 (0x6d82…ec4b), finalized #5446885 (0x1f31…4d80), ⬇ 8.0kiB/s ⬆ 8.0kiB/s
Jun 19 13:36:37 m-acala-02 acala[3326290]: 2024-06-19 13:36:37 [Relaychain] 💔 Ignored block (#21285723 -- 0x6360…8183) announcement from 12D3KooWJMffzdMYBu8PBd6CYnSyjwMNoFxBkNQGwrCbpkgp3FZm because all validation slots for this peer are occupied.
Jun 19 13:36:37 m-acala-02 acala[3326290]: 2024-06-19 13:36:37 [Relaychain] 💔 Ignored block (#21285724 -- 0xa8f1…b20c) announcement from 12D3KooWJMffzdMYBu8PBd6CYnSyjwMNoFxBkNQGwrCbpkgp3FZm because all validation slots for this peer are occupied.
Jun 19 13:36:40 m-acala-02 acala[3326290]: 2024-06-19 13:36:40 [Relaychain] ⚙️ Syncing 514.8 bps, target=#21283499 (40 peers), best: #4353847 (0x1a91…53c5), finalized #4353536 (0x4534…ee69), ⬇ 526.7kiB/s ⬆ 133.5kiB/s
Jun 19 13:36:40 m-acala-02 acala[3326290]: 2024-06-19 13:36:40 💤 Idle (9 peers), best: #6417399 (0x6d82…ec4b), finalized #5446885 (0x1f31…4d80), ⬇ 64 B/s ⬆ 51 B/s
Jun 19 13:36:40 m-acala-02 acala[3326290]: 2024-06-19 13:36:40 [Relaychain] Sending fatal alert BadCertificate
Jun 19 13:36:42 m-acala-02 acala[3326290]: 2024-06-19 13:36:42 ✨ Imported #6417400 (0x1d7a…c4f8)
Jun 19 13:36:45 m-acala-02 acala[3326290]: 2024-06-19 13:36:45 [Relaychain] ⚙️ Syncing 245.0 bps, target=#21283499 (40 peers), best: #4355072 (0x3c86…3229), finalized #4355072 (0x3c86…3229), ⬇ 180.4kiB/s ⬆ 31.5kiB/s
Jun 19 13:36:45 m-acala-02 acala[3326290]: 2024-06-19 13:36:45 💤 Idle (9 peers), best: #6417400 (0x1d7a…c4f8), finalized #5446885 (0x1f31…4d80), ⬇ 11.5kiB/s ⬆ 7.9kiB/s
Jun 19 13:36:49 m-acala-02 acala[3326290]: 2024-06-19 13:36:49 ✨ Imported #6417400 (0x7d55…3150)
Jun 19 13:36:50 m-acala-02 acala[3326290]: 2024-06-19 13:36:50 [Relaychain] ⚙️ Syncing 340.4 bps, target=#21283499 (40 peers), best: #4356774 (0x21c1…64e7), finalized #4356608 (0xb7b7…7d01), ⬇ 348.5kiB/s ⬆ 85.0kiB/s
Jun 19 13:36:50 m-acala-02 acala[3326290]: 2024-06-19 13:36:50 💤 Idle (9 peers), best: #6417400 (0x1d7a…c4f8), finalized #5446885 (0x1f31…4d80), ⬇ 31.0kiB/s ⬆ 10.8kiB/s
Jun 19 13:36:55 m-acala-02 acala[3326290]: 2024-06-19 13:36:55 [Relaychain] ⚙️ Syncing 274.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 185.2kiB/s ⬆ 32.0kiB/s
Jun 19 13:36:55 m-acala-02 acala[3326290]: 2024-06-19 13:36:55 💤 Idle (9 peers), best: #6417400 (0x1d7a…c4f8), finalized #5446885 (0x1f31…4d80), ⬇ 0.4kiB/s ⬆ 0.3kiB/s
Jun 19 13:37:00 m-acala-02 acala[3326290]: 2024-06-19 13:37:00 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 163.7kiB/s ⬆ 95.7kiB/s
Jun 19 13:37:00 m-acala-02 acala[3326290]: 2024-06-19 13:37:00 💤 Idle (9 peers), best: #6417400 (0x1d7a…c4f8), finalized #5446885 (0x1f31…4d80), ⬇ 0.4kiB/s ⬆ 0.3kiB/s
Jun 19 13:37:00 m-acala-02 acala[3326290]: 2024-06-19 13:37:00 ♻️ Reorg on #6417400,0x1d7a…c4f8 to #6417401,0xcf8c…2ac5, common ancestor #6417399,0x6d82…ec4b
Jun 19 13:37:00 m-acala-02 acala[3326290]: 2024-06-19 13:37:00 ✨ Imported #6417401 (0xcf8c…2ac5)
Jun 19 13:37:05 m-acala-02 acala[3326290]: 2024-06-19 13:37:05 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 73.8kiB/s ⬆ 35.9kiB/s
Jun 19 13:37:05 m-acala-02 acala[3326290]: 2024-06-19 13:37:05 💤 Idle (9 peers), best: #6417401 (0xcf8c…2ac5), finalized #5446885 (0x1f31…4d80), ⬇ 4.3kiB/s ⬆ 39.7kiB/s
Jun 19 13:37:10 m-acala-02 acala[3326290]: 2024-06-19 13:37:10 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 153.1kiB/s ⬆ 68.7kiB/s
Jun 19 13:37:10 m-acala-02 acala[3326290]: 2024-06-19 13:37:10 💤 Idle (9 peers), best: #6417401 (0xcf8c…2ac5), finalized #5446885 (0x1f31…4d80), ⬇ 0.1kiB/s ⬆ 70 B/s
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285719 -- 0x0aa7…72a2) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285720 -- 0x9c47…fb60) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285721 -- 0xf39e…48f3) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285721 -- 0x4304…351b) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285724 -- 0xa8f1…b20c) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285725 -- 0x6940…d82d) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285726 -- 0x9b12…437c) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:13 m-acala-02 acala[3326290]: 2024-06-19 13:37:13 ✨ Imported #6417402 (0x2cf2…0992)
Jun 19 13:37:15 m-acala-02 acala[3326290]: 2024-06-19 13:37:15 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 121.7kiB/s ⬆ 65.4kiB/s
Jun 19 13:37:15 m-acala-02 acala[3326290]: 2024-06-19 13:37:15 💤 Idle (9 peers), best: #6417402 (0x2cf2…0992), finalized #5446885 (0x1f31…4d80), ⬇ 7.7kiB/s ⬆ 22.1kiB/s
Jun 19 13:37:20 m-acala-02 acala[3326290]: 2024-06-19 13:37:20 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 76.7kiB/s ⬆ 52.1kiB/s
Jun 19 13:37:20 m-acala-02 acala[3326290]: 2024-06-19 13:37:20 💤 Idle (9 peers), best: #6417402 (0x2cf2…0992), finalized #5446885 (0x1f31…4d80), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
Jun 19 13:37:24 m-acala-02 acala[3326290]: 2024-06-19 13:37:24 ✨ Imported #6417403 (0xbd5a…8d09)
Jun 19 13:37:25 m-acala-02 acala[3326290]: 2024-06-19 13:37:25 [Relaychain] ⚙️ Syncing 407.8 bps, target=#21283499 (40 peers), best: #4360183 (0x06f8…4977), finalized #4359680 (0xa8c2…291c), ⬇ 353.5kiB/s ⬆ 86.7kiB/s
Jun 19 13:37:25 m-acala-02 acala[3326290]: 2024-06-19 13:37:25 💤 Idle (9 peers), best: #6417403 (0xbd5a…8d09), finalized #5446885 (0x1f31…4d80), ⬇ 21.7kiB/s ⬆ 7.6kiB/s
Jun 19 13:37:30 m-acala-02 acala[3326290]: 2024-06-19 13:37:30 [Relaychain] ⚙️ Syncing 478.4 bps, target=#21283499 (40 peers), best: #4362575 (0x5e61…3e97), finalized #4362240 (0xceeb…ccee), ⬇ 292.8kiB/s ⬆ 42.4kiB/s
Jun 19 13:37:30 m-acala-02 acala[3326290]: 2024-06-19 13:37:30 💤 Idle (9 peers), best: #6417403 (0xbd5a…8d09), finalized #5446885 (0x1f31…4d80), ⬇ 38 B/s ⬆ 38 B/s
Jun 19 13:37:35 m-acala-02 acala[3326290]: 2024-06-19 13:37:35 [Relaychain] ⚙️ Syncing 253.2 bps, target=#21283499 (40 peers), best: #4363841 (0x8b11…f7b2), finalized #4363776 (0x408a…c4ae), ⬇ 262.4kiB/s ⬆ 78.0kiB/s
Jun 19 13:37:35 m-acala-02 acala[3326290]: 2024-06-19 13:37:35 💤 Idle (9 peers), best: #6417403 (0xbd5a…8d09), finalized #5446885 (0x1f31…4d80), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
Jun 19 13:37:36 m-acala-02 acala[3326290]: 2024-06-19 13:37:36 ✨ Imported #6417403 (0x0c0c…6b39)
Jun 19 13:37:40 m-acala-02 acala[3326290]: 2024-06-19 13:37:40 [Relaychain] ⚙️ Syncing 331.8 bps, target=#21283499 (40 peers), best: #4365500 (0xfdb6…8396), finalized #4365312 (0x0242…16c5), ⬇ 201.7kiB/s ⬆ 38.4kiB/s
Jun 19 13:37:40 m-acala-02 acala[3326290]: 2024-06-19 13:37:40 💤 Idle (9 peers), best: #6417403 (0xbd5a…8d09), finalized #5446885 (0x1f31…4d80), ⬇ 4.2kiB/s ⬆ 18.5kiB/s
Jun 19 13:37:41 m-acala-02 acala[3326290]: 2024-06-19 13:37:41 [Relaychain] Sending fatal alert BadCertificate
Jun 19 13:37:45 m-acala-02 acala[3326290]: 2024-06-19 13:37:45 [Relaychain] ⚙️ Syncing 558.6 bps, target=#21283499 (40 peers), best: #4368293 (0x0938…608d), finalized #4367872 (0x94de…ddf6), ⬇ 494.3kiB/s ⬆ 122.0kiB/s
Jun 19 13:37:45 m-acala-02 acala[3326290]: 2024-06-19 13:37:45 💤 Idle (9 peers), best: #6417403 (0xbd5a…8d09), finalized #5446885 (0x1f31…4d80), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
Jun 19 13:37:48 m-acala-02 acala[3326290]: 2024-06-19 13:37:48 ♻️ Reorg on #6417403,0xbd5a…8d09 to #6417404,0x844e…d5e4, common ancestor #6417402,0x2cf2…0992
Jun 19 13:37:48 m-acala-02 acala[3326290]: 2024-06-19 13:37:48 ✨ Imported #6417404 (0x844e…d5e4)
Jun 19 13:37:50 m-acala-02 acala[3326290]: 2024-06-19 13:37:50 [Relaychain] ⚙️ Syncing 120.6 bps, target=#21283499 (40 peers), best: #4368896 (0x4d9f…a405), finalized #4368896 (0x4d9f…a405), ⬇ 120.7kiB/s ⬆ 46.3kiB/s
Jun 19 13:37:50 m-acala-02 acala[3326290]: 2024-06-19 13:37:50 💤 Idle (9 peers), best: #6417404 (0x844e…d5e4), finalized #5446885 (0x1f31…4d80), ⬇ 7.9kiB/s ⬆ 11.4kiB/s
Jun 19 13:37:55 m-acala-02 acala[3326290]: 2024-06-19 13:37:55 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4368896 (0x4d9f…a405), finalized #4368896 (0x4d9f…a405), ⬇ 162.3kiB/s ⬆ 79.3kiB/s
Jun 19 13:37:55 m-acala-02 acala[3326290]: 2024-06-19 13:37:55 💤 Idle (9 peers), best: #6417404 (0x844e…d5e4), finalized #5446885 (0x1f31…4d80), ⬇ 64 B/s ⬆ 64 B/s
Jun 19 13:38:00 m-acala-02 acala[3326290]: 2024-06-19 13:38:00 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4368896 (0x4d9f…a405), finalized #4368896 (0x4d9f…a405), ⬇ 25.7kiB/s ⬆ 21.3kiB/s
Jun 19 13:38:00 m-acala-02 acala[3326290]: 2024-06-19 13:38:00 💤 Idle (9 peers), best: #6417404 (0x844e…d5e4), finalized #5446885 (0x1f31…4d80), ⬇ 38 B/s ⬆ 38 B/s
Jun 19 13:38:01 m-acala-02 acala[3326290]: 2024-06-19 13:38:01 ✨ Imported #6417405 (0xe20f…3572)
Jun 19 13:38:05 m-acala-02 acala[3326290]: 2024-06-19 13:38:05 [Relaychain] ⚙️ Syncing 171.4 bps, target=#21283499 (40 peers), best: #4369753 (0xb78e…8959), finalized #4369408 (0xc265…54ba), ⬇ 245.2kiB/s ⬆ 79.9kiB/s
Jun 19 13:38:05 m-acala-02 acala[3326290]: 2024-06-19 13:38:05 💤 Idle (9 peers), best: #6417405 (0xe20f…3572), finalized #5446885 (0x1f31…4d80), ⬇ 11.4kiB/s ⬆ 7.9kiB/s
Jun 19 13:38:10 m-acala-02 acala[3326290]: 2024-06-19 13:38:10 [Relaychain] ⚙️ Syncing 341.4 bps, target=#21283499 (40 peers), best: #4371460 (0x9e0f…7803), finalized #4371456 (0x4c13…10de), ⬇ 259.3kiB/s ⬆ 32.9kiB/s
Jun 19 13:38:10 m-acala-02 acala[3326290]: 2024-06-19 13:38:10 💤 Idle (9 peers), best: #6417405 (0xe20f…3572), finalized #5446885 (0x1f31…4d80), ⬇ 64 B/s ⬆ 64 B/s
Jun 19 13:38:12 m-acala-02 acala[3326290]: 2024-06-19 13:38:12 ✨ Imported #6417406 (0x5527…76fe)
Jun 19 13:38:15 m-acala-02 acala[3326290]: 2024-06-19 13:38:15 [Relaychain] ⚙️ Syncing 445.4 bps, target=#21283499 (40 peers), best: #4373687 (0xa40a…3b2e), finalized #4373504 (0x24a3…9229), ⬇ 335.4kiB/s ⬆ 59.1kiB/s
Jun 19 13:38:15 m-acala-02 acala[3326290]: 2024-06-19 13:38:15 💤 Idle (9 peers), best: #6417406 (0x5527…76fe), finalized #5446885 (0x1f31…4d80), ⬇ 21.7kiB/s ⬆ 7.5kiB/s
Jun 19 13:38:20 m-acala-02 acala[3326290]: 2024-06-19 13:38:20 [Relaychain] ⚙️ Syncing 488.8 bps, target=#21283499 (40 peers), best: #4376136 (0x3ca5…5ad5), finalized #4376064 (0xdd54…c925), ⬇ 271.8kiB/s ⬆ 24.9kiB/s
Jun 19 13:38:20 m-acala-02 acala[3326290]: 2024-06-19 13:38:20 💤 Idle (9 peers), best: #6417406 (0x5527…76fe), finalized #5446885 (0x1f31…4d80), ⬇ 25 B/s ⬆ 25 B/s
Jun 19 13:38:24 m-acala-02 acala[3326290]: 2024-06-19 13:38:24 ✨ Imported #6417407 (0xb3fa…79b2)
Jun 19 13:38:25 m-acala-02 acala[3326290]: 2024-06-19 13:38:25 [Relaychain] ⚙️ Syncing 395.2 bps, target=#21283499 (40 peers), best: #4378112 (0x4d81…be77), finalized #4378112 (0x4d81…be77), ⬇ 312.5kiB/s ⬆ 56.0kiB/s
Jun 19 13:38:25 m-acala-02 acala[3326290]: 2024-06-19 13:38:25 💤 Idle (9 peers), best: #6417407 (0xb3fa…79b2), finalized #5446885 (0x1f31…4d80), ⬇ 3.9kiB/s ⬆ 28.9kiB/s
Jun 19 13:38:27 m-acala-02 acala[3326290]: 2024-06-19 13:38:27 [Relaychain] Sending fatal alert BadCertificate
Jun 19 13:38:30 m-acala-02 acala[3326290]: 2024-06-19 13:38:30 [Relaychain] ⚙️ Syncing 289.0 bps, target=#21283499 (40 peers), best: #4379557 (0xf518…c896), finalized #4379136 (0x28d0…864b), ⬇ 192.9kiB/s ⬆ 47.6kiB/s
Jun 19 13:38:30 m-acala-02 acala[3326290]: 2024-06-19 13:38:30 💤 Idle (9 peers), best: #6417407 (0xb3fa…79b2), finalized #5446885 (0x1f31…4d80), ⬇ 5.1kiB/s ⬆ 2.2kiB/s
Jun 19 13:38:32 m-acala-02 acala[3326290]: 2024-06-19 13:38:32 [Relaychain] Sending fatal alert BadCertificate
Jun 19 13:38:35 m-acala-02 acala[3326290]: 2024-06-19 13:38:35 [Relaychain] ⚙️ Syncing 427.8 bps, target=#21283499 (40 peers), best: #4381696 (0x0d04…3e90), finalized #4381696 (0x0d04…3e90), ⬇ 356.4kiB/s ⬆ 73.0kiB/s
Jun 19 13:38:35 m-acala-02 acala[3326290]: 2024-06-19 13:38:35 💤 Idle (9 peers), best: #6417407 (0xb3fa…79b2), finalized #5446885 (0x1f31…4d80), ⬇ 0.3kiB/s ⬆ 0.3kiB/s
Jun 19 13:38:37 m-acala-02 acala[3326290]: 2024-06-19 13:38:37 ✨ Imported #6417408 (0xd7c9…528c)
Jun 19 13:38:40 m-acala-02 acala[3326290]: 2024-06-19 13:38:40 [Relaychain] ⚙️ Syncing 409.6 bps, target=#21283499 (40 peers), best: #4383744 (0x569e…c117), finalized #4383744 (0x569e…c117), ⬇ 325.3kiB/s ⬆ 54.9kiB/s
Jun 19 13:38:40 m-acala-02 acala[3326290]: 2024-06-19 13:38:40 💤 Idle (9 peers), best: #6417408 (0xd7c9…528c), finalized #5446885 (0x1f31…4d80), ⬇ 12.6kiB/s ⬆ 22.5kiB/s
Jun 19 13:38:45 m-acala-02 acala[3326290]: 2024-06-19 13:38:45 [Relaychain] ⚙️ Syncing 0.0 bps, target=#21283499 (40 peers), best: #4383744 (0x569e…c117), finalized #4383744 (0x569e…c117), ⬇ 91.4kiB/s ⬆ 45.3kiB/s
Jun 19 13:38:45 m-acala-02 acala[3326290]: 2024-06-19 13:38:45 💤 Idle (9 peers), best: #6417408 (0xd7c9…528c), finalized #5446885 (0x1f31…4d80), ⬇ 3.5kiB/s ⬆ 1.9kiB/s
I see a few message like Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Example:
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -1024 to -2147483648. Reason: same small block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
No, it didn't help. I tried multiple times to restart the node.
Now, I deleted the acala database and the node is syncing from genesis again... I will update this issue...
Do you still have the database around? Restarting with debug or even trace logs might reveal what is going on.
RUST_LOG=debug /your/collator/binary --args-you-normally-pass
Trace for everything is likely to spammy, but only for sync should work:
RUST_LOG=sync=trace /your/collator/binary --args-you-normally-pass
I will enable that level of logging if I see this or any other issues. Thanks