rootzoll
1.11.4 from scratch: does not complete initial blockchain sync on my raspberry pi4
Hi,
I have been trying to get raspiblitz to get running for a long time now on my. This especially due to the enormous time it takes to get the initial blockchain synced.
And each time it ends up after days/weeks in a sync restart loop. (current try, is at 1325 restarts)
During the months I tried, I updated raspiblitz every time a new version came out, hoping something would improve. Also tried several times from scratch by formatting the udb ssd disk.
The only reason I keep on trying it because I have a raspiblitz wallet (c-lightning) with some credit. Tried op access the wallet credit with other tools, but have not found any working yet.
Steps to reproduce the behavior:
Flash software to SD, format SSD disk start installation process.
https://termbin.com/1rhh
2024-11-29T08:47:46Z Verification progress: 83% 2024-11-29T08:48:45Z Verification progress: 99% 2024-11-29T08:48:45Z Verification: No coin database inconsistencies in last 6 blocks (16834 transactions) 2024-11-29T08:48:45Z block index 430803ms 2024-11-29T08:48:45Z Setting NODE_NETWORK on non-prune mode 2024-11-29T08:48:45Z block tree size = 871939 2024-11-29T08:48:45Z nBestHeight = 564916 2024-11-29T08:48:45Z initload thread start 2024-11-29T08:48:45Z Bound to 127.0.0.1:8333 2024-11-29T08:48:45Z Bound to 127.0.0.1:8334 2024-11-29T08:48:45Z ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large: iostream error at FlatFilePos(nFile=1543, nPos=24286267) 2024-11-29T08:48:45Z *** Failed to read block 2024-11-29T08:48:45Z Error: A fatal internal error occurred, see debug.log for details 2024-11-29T08:48:45Z Loaded 0 addresses from "anchors.dat" 2024-11-29T08:48:45Z *** Failed to connect best block (Failed to read block) 2024-11-29T08:48:45Z Error: A fatal internal error occurred, see debug.log for details 2024-11-29T08:48:45Z 0 block-relay-only anchors will be tried for connections. 2024-11-29T08:48:45Z init message: Starting network threads… 2024-11-29T08:48:45Z torcontrol thread start 2024-11-29T08:48:45Z DNS seeding disabled 2024-11-29T08:48:45Z [tor] Reading cached private key from /mnt/hdd/bitcoin/onion_v3_private_key 2024-11-29T08:48:45Z [tor] Successfully connected! 2024-11-29T08:48:45Z init message: Done loading 2024-11-29T08:48:45Z tor: Thread interrupt 2024-11-29T08:48:45Z addcon thread start 2024-11-29T08:48:45Z msghand thread start 2024-11-29T08:48:45Z msghand thread exit 2024-11-29T08:48:45Z Shutdown: In progress... 2024-11-29T08:48:45Z i2paccept thread start 2024-11-29T08:48:45Z i2paccept thread exit 2024-11-29T08:48:45Z addcon thread exit 2024-11-29T08:48:45Z opencon thread start 2024-11-29T08:48:45Z opencon thread exit 2024-11-29T08:48:45Z net thread start 2024-11-29T08:48:45Z net thread exit 2024-11-29T08:48:45Z Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast 2024-11-29T08:48:45Z initload thread exit 2024-11-29T08:48:45Z torcontrol thread exit 2024-11-29T08:48:45Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat started 2024-11-29T08:48:45Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat completed (0.01s) 2024-11-29T08:48:45Z scheduler thread exit 2024-11-29T08:48:45Z Flushed fee estimates to fee_estimates.dat. 2024-11-29T08:48:46Z Shutdown: done
Optimizing the initial sync while also implementing the new Assume UTXO feature is one goal for our upcomming v1.12.0 release - so if you can wait a bit longer, I would recommend waiting for that version.
In the meantime I made progress by replacing my 1TB SSD disk and power adaptor. Also upgraded to v1.11.4. At least the btc blockchain sync works.
Now I don't get past the lightning sync.
It seems the lightningd backup plugin is filling up my 32GB SD. should that not we written to the SSD disk?
-rw-r--r-- 1 bitcoin bitcoin 3.7G Jan 9 01:33 /home/bitcoin/lightningd.sqlite3.backup
2025-01-09T01:33:55.364Z **BROKEN** lightningd: Plugin '/home/bitcoin/cl-plugins-available/plugins/backup/backup.py' returned an invalid response to the db_write hook: {"jsonrpc": "2.0", "id": "cln:db_write#114773", "error": {"code": -32600, "message": "Error while processing db_write: [Errno 28] No space left on device", "traceback": "Traceback (most recent call last):\n File \"/home/bitcoin/.local/lib/python3.11/site-packages/pyln/client/plugin.py\", line 705, in _dispatch_request\n result = self._exec_func(method.func, request)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/bitcoin/.local/lib/python3.11/site-packages/pyln/client/plugin.py\", line 685, in _exec_func\n ret = func(*ba.args, **ba.kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/bitcoin/cl-plugins-available/plugins/backup/backup.py\", line 66, in on_db_write\n if plugin.backend.add_change(change):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/bitcoin/cl-plugins-available/plugins/backup/filebackend.py\", line 78, in add_change\n f.write(payload)\nOSError: [Errno 28] No space left on device\n"}}
2025-01-09T01:33:55.944Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v24.11)
disabling the lightningd backup plugin prevented at least my SD to fillup.
Now both btc and lightning is 100% synced, but new issue. I will reboot and see if it gets beyond this point.
2025-01-11T04:53:04.335Z UNUSUAL plugin-bcli: bitcoin-cli: finished bitcoin-cli -rpcclienttimeout=60 getblockhash 878733 (17663 ms)
2025-01-11T04:55:04.766Z UNUSUAL plugin-bcli: bitcoin-cli: finished bitcoin-cli -rpcclienttimeout=60 getmempoolinfo (60180 ms)
2025-01-11T04:55:04.803Z UNUSUAL lightningd: Unable to estimate any fees
2025-01-11T04:55:09.426Z UNUSUAL plugin-bcli: bitcoin-cli: finished bitcoin-cli -rpcclienttimeout=60 getblockhash 878734 (60138 ms)
2025-01-11T04:55:09.438Z UNUSUAL plugin-bcli: bitcoin-cli -rpcclienttimeout=60 getblockhash 878734 exited with status 1
2025-01-11T04:55:40.250Z UNUSUAL plugin-bcli: bitcoin-cli: finished bitcoin-cli -rpcclienttimeout=60 getblockhash 878734 (29812 ms)
2025-01-11T04:56:43.136Z UNUSUAL plugin-bcli: bitcoin-cli: finished bitcoin-cli -rpcclienttimeout=60 getblockhash 878735 (60132 ms)
2025-01-11T04:56:43.187Z UNUSUAL plugin-bcli: bitcoin-cli -rpcclienttimeout=60 getblockhash 878735 exited with status 1
2025-01-11T04:57:10.742Z UNUSUAL plugin-bcli: bitcoin-cli: finished bitcoin-cli -rpcclienttimeout=60 getmempoolinfo (60249 ms)
2025-01-11T04:57:10.786Z UNUSUAL lightningd: Unable to estimate any fees
2025-01-11T04:57:44.365Z UNUSUAL plugin-bcli: bitcoin-cli: finished bitcoin-cli -rpcclienttimeout=60 getblockhash 878735 (60142 ms)
2025-01-11T04:57:44.406Z **BROKEN** plugin-bcli: bitcoin-cli -rpcclienttimeout=60 getblockhash 878735 exited 1 (after 1 other errors) 'error: timeout on transient error: Could not connect to the server 127.0.0.1:8332 (error code 0 - \"timeout reached\")\n\nMake sure the bitcoind server is running and that you are connecting to the correct RPC port.\n'; we have been retrying command for --bitcoin-retry-timeout=60 seconds; bitcoind setup or our --bitcoin-* configs broken?
2025-01-11T04:57:44.424Z INFO plugin-bcli: Killing plugin: exited during normal operation
2025-01-11T04:57:44.438Z **BROKEN** lightningd: The Bitcoin backend died.
2025-01-11T04:57:45.039Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v24.11)
2025-01-11T04:57:45.040Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x55731d0537
2025-01-11T04:57:45.041Z **BROKEN** lightningd: backtrace: common/daemon.c:75 (crashdump) 0x55731d058b
2025-01-11T04:57:45.041Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7fa826c7af
2025-01-11T04:57:45.042Z **BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:44 (__pthread_kill_implementation) 0x7fa7e30a50
2025-01-11T04:57:45.042Z **BROKEN** lightningd: backtrace: ../sysdeps/posix/raise.c:26 (__GI_raise) 0x7fa7dea72b
2025-01-11T04:57:45.042Z **BROKEN** lightningd: backtrace: ./stdlib/abort.c:79 (__GI_abort) 0x7fa7dd747b
2025-01-11T04:57:45.042Z **BROKEN** lightningd: backtrace: lightningd/log.c:1038 (fatal_vfmt) 0x5573174bf3
2025-01-11T04:57:45.043Z **BROKEN** lightningd: backtrace: lightningd/log.c:1048 (fatal) 0x5573174c83
2025-01-11T04:57:45.045Z **BROKEN** lightningd: backtrace: lightningd/bitcoind.c:36 (bitcoin_destructor) 0x5573143acb
2025-01-11T04:57:45.046Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:246 (notify) 0x55732c47e7
2025-01-11T04:57:45.047Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:437 (del_tree) 0x55732c48d3
2025-01-11T04:57:45.047Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:532 (tal_free) 0x55732c4ee7
2025-01-11T04:57:45.047Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:459 (plugin_kill) 0x557319a4db
2025-01-11T04:57:45.048Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:890 (plugin_conn_finish) 0x557319a7ff
2025-01-11T04:57:45.048Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:246 (destroy_conn) 0x55732b91db
2025-01-11T04:57:45.048Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:252 (destroy_conn_close_fd) 0x55732b9207
2025-01-11T04:57:45.049Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:246 (notify) 0x55732c47e7
2025-01-11T04:57:45.049Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:437 (del_tree) 0x55732c48d3
2025-01-11T04:57:45.049Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:532 (tal_free) 0x55732c4ee7
2025-01-11T04:57:45.050Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:496 (io_close) 0x55732b790f
2025-01-11T04:57:45.050Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:415 (do_plan) 0x55732b79df
2025-01-11T04:57:45.050Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x55732b7a4f
2025-01-11T04:57:45.056Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:455 (io_loop) 0x55732b9983
2025-01-11T04:57:45.057Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x557316c183
2025-01-11T04:57:45.058Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1484 (main) 0x5573172057
2025-01-11T04:57:45.058Z **BROKEN** lightningd: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x7fa7dd773f
2025-01-11T04:57:45.058Z **BROKEN** lightningd: backtrace: ../csu/libc-start.c:360 (__libc_start_main_impl) 0x7fa7dd7817
2025-01-11T04:57:45.058Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x55731423ef
2025-01-11T04:57:45.059Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
I don't seem to get beyond btc/lightning syncing. Left it for 24h at 100%/100% ,but nothing happened. Rebooted, but and up in the same state.
Any suggestion is welcome.
Don't see any helpful things in the logs, expect some i2p errors.
/mnt/hdd/bitcoin/debug.log
2025-01-15T08:42:00Z New manual v1 peer connected: version: 70016, blocks=879341, peer=903
2025-01-15T08:45:00Z [i2p] Error connecting to 7l6l47l7pwi7vfe7romlkpnoqu76dr4gphrdmic4tzdkgiifgz5q.b32.i2p:0: "STREAM STATUS RESULT=CANT_REACH_PEER MESSAGE="LeaseSet not found""
2025-01-15T08:45:30Z [i2p] Error connecting to bl5njwu5fcjfqztfvotuf54t5kza6raqvtzopypbjvzxgi5c672q.b32.i2p:0: Unexpected reply to "NAMING LOOKUP NAME=bl5njwu5fcjfqztfvotuf54t5kza6raqvtzopypbjvzxgi5c672q.b32.i2p": "NAMING REPLY RESULT=INVALID_KEY NAME=bl5njwu5fcjfqztfvotuf54t5kza6raqvtzopypbjvzxgi5c672q.b32.i2p"
2025-01-15T08:45:44Z [i2p] Error connecting to 4fa7d27nlhf4uuuvxuenc2p6ef6lflgswmt5rvcpwt36j7vq35fa.b32.i2p:0: Unexpected reply to "NAMING LOOKUP NAME=4fa7d27nlhf4uuuvxuenc2p6ef6lflgswmt5rvcpwt36j7vq35fa.b32.i2p": "NAMING REPLY RESULT=INVALID_KEY NAME=4fa7d27nlhf4uuuvxuenc2p6ef6lflgswmt5rvcpwt36j7vq35fa.b32.i2p"
2025-01-15T08:46:13Z [i2p] Error connecting to rtwm6njef5tifwv5ols3b4nsboniig5kdw32ik5qbv3wmhoo3pba.b32.i2p:0: Unexpected reply to "NAMING LOOKUP NAME=rtwm6njef5tifwv5ols3b4nsboniig5kdw32ik5qbv3wmhoo3pba.b32.i2p": "NAMING REPLY RESULT=INVALID_KEY NAME=rtwm6njef5tifwv5ols3b4nsboniig5kdw32ik5qbv3wmhoo3pba.b32.i2p"
2025-01-15T08:46:18Z New block-relay-only v1 peer connected: version: 70016, blocks=879341, peer=906
2025-01-15T08:47:13Z [i2p] Error connecting to 2ny6ifcupp7xx75a2anecwhih5kibumg2o3uqdk5j2ejn26jkv4q.b32.i2p:0: Unexpected reply to "NAMING LOOKUP NAME=2ny6ifcupp7xx75a2anecwhih5kibumg2o3uqdk5j2ejn26jkv4q.b32.i2p": "NAMING REPLY RESULT=INVALID_KEY NAME=2ny6ifcupp7xx75a2anecwhih5kibumg2o3uqdk5j2ejn26jkv4q.b32.i2p"
2025-01-15T08:47:28Z [i2p] Error connecting to vkz72iimesw6klnurujjyoyaphgxfj74wg7hjuiyr3jhqu7fo3rq.b32.i2p:0: "STREAM STATUS RESULT=CANT_REACH_PEER MESSAGE="LeaseSet not found""
2025-01-15T08:48:32Z [i2p] Error connecting to s4qlly4iwzevejk4ex5zfqhb4t666o73mvdmy4gpu47tpb775nja.b32.i2p:0: Unexpected reply to "NAMING LOOKUP NAME=s4qlly4iwzevejk4ex5zfqhb4t666o73mvdmy4gpu47tpb775nja.b32.i2p": "NAMING REPLY RESULT=INVALID_KEY NAME=s4qlly4iwzevejk4ex5zfqhb4t666o73mvdmy4gpu47tpb775nja.b32.i2p"
2025-01-15T08:48:42Z [i2p] Error connecting to c5fk6xjwbfdurepihkdjvligexlzwu64zh2e5sjd5pmhvv34kmaq.b32.i2p:0: Unexpected reply to "NAMING LOOKUP NAME=c5fk6xjwbfdurepihkdjvligexlzwu64zh2e5sjd5pmhvv34kmaq.b32.i2p": "NAMING REPLY RESULT=INVALID_KEY NAME=c5fk6xjwbfdurepihkdjvligexlzwu64zh2e5sjd5pmhvv34kmaq.b32.i2p"
/mnt/hdd/app-data/.lightning/bitcoin/cl.log
2025-01-14T16:09:55.113Z INFO lightningd: v24.11
2025-01-14T16:10:02.578Z UNUSUAL plugin-bookkeeper: topic 'utxo_deposit' is not a known notification topic
2025-01-14T16:10:02.578Z UNUSUAL plugin-bookkeeper: topic 'utxo_spend' is not a known notification topic
2025-01-14T16:11:15.176Z INFO connectd: Static Tor service onion address: "sphkwdkyfvhnusvbgf7hv6fgfz6chduav4pwsgzdyxfb4kcxjsiibqqd.onion:9736,127.0.0.1:9736" bound from extern port 9736
2025-01-14T16:11:16.679Z INFO plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2025-01-14T16:11:36.731Z UNUSUAL plugin-bcli: bitcoin-cli: finished bitcoin-cli -rpcclienttimeout=60 estimatesmartfee 2 CONSERVATIVE (10397 ms)
2025-01-14T16:12:41.365Z INFO lightningd: --------------------------------------------------
2025-01-14T16:12:41.366Z INFO lightningd: Server started with public key 03a40a68a867f199eb0f87a9d3883f96ecb0a8dcf946c95b993a272c14c18aadeb, alias SLIMYBOUNCE (color #03a40a) and lightningd v24.11
2025-01-14T16:12:41.668Z INFO plugin-clnrest: REST server running at https://0.0.0.0:7378
@jalbstmeijer having 0 peers with CLN could be a reason that it does not finish. Not sure why - if you have time leave it off until next release is available (with new CLN verison) and then try again.
TODO: Rerun TEST BTC+CLN complete IBD