rootzoll
CLN – cl.backup.sh: line 142: echo: write error: No space left on device
Describe the bug Core Lightning isn’t syncing and stuck at 98% since two days. I tried several times restarting bitcoind and/or lightningd.service. I’ll get always the same result:
`● lightningd.service - c-lightning daemon on mainnet Loaded: loaded (/etc/systemd/system/lightningd.service; enabled; preset: enabled) Active: active (running) since Wed 2024-12-11 11:23:36 GMT; 445ms ago Process: 254827 ExecStartPre=/home/admin/config.scripts/cl.check.sh prestart mainnet (code=exited, status=0/SUCCESS) Process: 254844 ExecStartPost=/home/admin/config.scripts/cl.check.sh poststart mainnet (code=exited, status=0/SUCCESS) Main PID: 254845 Tasks: 2 (limit: 9275) Memory: 127.3M CPU: 4.678s CGroup: /system.slice/lightningd.service ├─254968 python3 /usr/local/libexec/c-lightning/plugins/clnrest/clnrest.py └─254970 python3 /home/bitcoin/cl-plugins-available/plugins/backup/backup.py
Dec 11 11:23:29 raspberrypi systemd[1]: lightningd.service: Can't open PID file /run/lightningd/lightningd.pid (yet?) after start-post: No such file or directory Dec 11 11:23:36 raspberrypi systemd[1]: lightningd.service: Supervising process 254845 which is not our child. We'll most likely not notice when it exits. Dec 11 11:23:36 raspberrypi systemd[1]: Started lightningd.service - c-lightning daemon on mainnet. Dec 11 11:23:36 raspberrypi sh[254845]: Plugin '/home/bitcoin/cl-plugins-available/plugins/backup/backup.py' returned an invalid response to the db_write hook: {"jsonrpc": "2.0", "id": "cln:db_write#18", "error": {"code": -32600, "message": "Error while processing db_write: [Errno 28] No space left on device", "traceback": "Traceback (most recen> Dec 11 11:23:36 raspberrypi sh[254970]: object address : 0x7f8a4d5540 Dec 11 11:23:36 raspberrypi sh[254970]: object refcount : 2 Dec 11 11:23:36 raspberrypi sh[254970]: object type : 0xa10b00 Dec 11 11:23:36 raspberrypi sh[254970]: object type name: BrokenPipeError Dec 11 11:23:36 raspberrypi sh[254970]: object repr : BrokenPipeError(32, 'Broken pipe') Dec 11 11:23:36 raspberrypi sh[254970]: lost sys.stderr`
If I try a backup, I’ll get this:
Removed "/etc/systemd/system/multi-user.target.wants/lightningd.service". Created symlink /etc/systemd/system/multi-user.target.wants/lightningd.service → /etc/systemd/system/lightningd.service. shred: /dev/shm/.cl.pw: pass 1/4 (random)... shred: /dev/shm/.cl.pw: pass 2/4 (random)... shred: /dev/shm/.cl.pw: pass 3/4 (random)... shred: /dev/shm/.cl.pw: pass 4/4 (000000)... shred: /dev/shm/.cl.pw: removing shred: /dev/shm/.cl.pw: renamed to /dev/shm/000000 shred: /dev/shm/000000: renamed to /dev/shm/00000 shred: /dev/shm/00000: renamed to /dev/shm/0000 shred: /dev/shm/0000: renamed to /dev/shm/000 shred: /dev/shm/000: renamed to /dev/shm/00 shred: /dev/shm/00: renamed to /dev/shm/0 shred: /dev/shm/.cl.pw: removed /home/admin/config.scripts/cl.backup.sh: line 142: echo: write error: No space left on device tar: Removing leading /' from member names
/mnt/hdd/app-data/.lightning/
/mnt/hdd/app-data/.lightning/version.info
/mnt/hdd/app-data/.lightning/bitcoin/
/mnt/hdd/app-data/.lightning/bitcoin/crash.log.20241211083720
/mnt/hdd/app-data/.lightning/bitcoin/crash.log.20241211120257
/mnt/hdd/app-data/.lightning/bitcoin/backup.lock
/mnt/hdd/app-data/.lightning/bitcoin/lightningd.sqlite3
/mnt/hdd/app-data/.lightning/bitcoin/emergency.recover
/mnt/hdd/app-data/.lightning/bitcoin/summars/
/mnt/hdd/app-data/.lightning/bitcoin/summars/availdb.json
/mnt/hdd/app-data/.lightning/bitcoin/gossip_store2.corrupt
/mnt/hdd/app-data/.lightning/bitcoin/crash.log.20241210144732
/mnt/hdd/app-data/.lightning/bitcoin/hsm_secret
tar: /mnt/hdd/app-data/.lightning/bitcoin/lightning-rpc: socket ignored
/mnt/hdd/app-data/.lightning/bitcoin/gossip_store
/mnt/hdd/app-data/.lightning/bitcoin/crash.log.20241210071337
/mnt/hdd/app-data/.lightning/bitcoin/cl.log_old/
/mnt/hdd/app-data/.lightning/bitcoin/seedwords.info
/mnt/hdd/app-data/.lightning/bitcoin/gossip_store3.corrupt
/mnt/hdd/app-data/.lightning/bitcoin/crash.log.20241211092313
/mnt/hdd/app-data/.lightning/bitcoin/cl.log
/mnt/hdd/app-data/.lightning/bitcoin/gossip_store.corrupt
/mnt/hdd/app-data/.lightning/bitcoin/lightningd.sqlite3-journal
/mnt/hdd/app-data/.lightning/bitcoin/accounts.sqlite3
/mnt/hdd/app-data/.lightning/config
/home/admin/config.scripts/cl.backup.sh: line 193: echo: write error: No space left on device
/home/admin/config.scripts/cl.backup.sh: line 194: echo: write error: No space left on device
/home/admin/config.scripts/cl.backup.sh: line 195: echo: write error: No space left on device
/home/admin/config.scripts/cl.backup.sh: line 196: echo: write error: No space left on device
/home/admin/config.scripts/cl.backup.sh: line 197: echo: write error: No space left on device
/home/admin/config.scripts/cl.backup.sh: line 198: echo: write error: No space left on device
/home/admin/config.scripts/cl.backup.sh: line 199: echo: write error: No space left on device
/home/admin/config.scripts/cl.backup.sh: line 200: echo: write error: No space left on device
/home/admin/config.scripts/cl.backup.sh: line 201: echo: write error: No space left on device
/home/admin/99clRepairMenu.sh: line 118: echo: write error: No space left on device
/home/admin/99clRepairMenu.sh: line 119: echo: write error: No space left on device`
Any idea, if there is a chance solving this problem without loosing my funds?
To Reproduce Steps to reproduce the behavior.
Debug file If I try to make a debug-log, I’ll get this:
/home/admin/98repairMenu.sh: line 101: echo: write error: No space left on device
`# waiting to unlock wallet (0) ...
waiting to unlock wallet (5) ...
waiting to unlock wallet (10) ...
waiting to unlock wallet (15) ...
waiting to unlock wallet (20) ...
waiting to unlock wallet (25) ...
waiting to unlock wallet (30) ...
waiting to unlock wallet (35) ...
waiting to unlock wallet (40) ...
waiting to unlock wallet (45) ...
waiting to unlock wallet (50) ...
waiting to unlock wallet (55) ...
Failed to unlock the lightningd wallet - giving up after 1 minute
The last lines of the lightningd logs ('sudo tail -n 5 /home/bitcoin/.lightning/bitcoin/cl.log'):
2024-12-11T15:39:30.120Z INFO lightningd: v24.02.1 2024-12-11T15:39:50.315Z INFO plugin-backup.py: root : Comparing backup version 984907 versus first write version 984908 2024-12-11T15:39:50.315Z INFO plugin-backup.py: Comparing backup version 984907 versus first write version 984908 2024-12-11T15:39:50.315Z INFO plugin-backup.py: root : Versions match up 2024-12-11T15:39:50.315Z INFO plugin-backup.py: Versions match up
The last lines of the lightningd journal ('sudo journalctl -u lightningd'):
Dec 11 15:39:50 raspberrypi sh[1691119]: Plugin '/home/bitcoin/cl-plugins-available/plugins/backup/backup.py' returned an invalid response to the db_write hook: {"jsonrpc": > Dec 11 15:39:50 raspberrypi systemd[1]: lightningd.service: Killing process 1691119 (n/a) with signal SIGKILL. Dec 11 15:39:50 raspberrypi systemd[1]: lightningd.service: Killing process 1691119 (n/a) with signal SIGKILL. Dec 11 15:39:50 raspberrypi systemd[1]: lightningd.service: Deactivated successfully. Dec 11 15:39:50 raspberrypi systemd[1]: lightningd.service: Consumed 17.111s CPU time.`
There is still plenty of space at the device for sure, used around 46% till now on it ...
No space left on device sounds like your sd card or SSD is full.
You may post the *** MOUNTED DRIVES *** & *** SD CARD HOMES *** section form the debug report so that we can look a bit more detailed what has eaten your space there.
But in general - check if you can delete any other apps installed you dont need. Next step would be to get a bigger sd card and reflash it with the latest RaspiBlitz and run a recovery. Your funds are stored the SSD - that also could be full (and should be at least 1TB big), but we check that last.
still strugling with my lightning-node. meanwhile i closed my channel. should I uninstall lightningd and reinstall it again? if so, will I get my on-chain-funds back afterwards? and how to uninstall and reinstall lightningd on raspiblitz?
here is my debug-log:
https://termbin.com/5bzj
lightning still not starting anymore:
lightningd.service - c-lightning daemon on mainnet Loaded: loaded (/etc/systemd/system/lightningd.service; enabled; preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Mon 2025-01-27 11:33:53 GMT; 6s ago Process: 1020289 ExecStartPre=/home/admin/config.scripts/cl.check.sh prestart mainnet (code=exited, status=0/SUCCESS) Process: 1020307 ExecStart=/bin/sh -c /usr/local/bin/lightningd --conf=/home/bitcoin/.lightning/config --pid-file=/run/lightningd/lightningd.pid --rpc-file-mode 0660 (code=exited, status=1/FAILURE) Process: 1020308 ExecStartPost=/home/admin/config.scripts/cl.check.sh poststart mainnet (code=exited, status=0/SUCCESS) Main PID: 1020307 (code=exited, status=1/FAILURE) CPU: 2.718s
can’t recover my on-chanin-funds till now, because recovering with the seed ends also in exit-code-issues ...
@rootzoll can I fix this or should I reinstall c-lightning? if so, how can I uninstall and reinstall it and is it afterwards still possible to get my on-chain-funds back?
So first of all your free space on the system looks fine .. not sure where that came from:
*** MOUNTED DRIVES ***
df -T -h
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs tmpfs 781M 1.2M 780M 1% /run
/dev/mmcblk0p2 ext4 29G 17G 12G 59% /
tmpfs tmpfs 3.9G 1.1M 3.9G 1% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 32M 24K 32M 1% /var/cache/raspiblitz
/dev/mmcblk0p1 vfat 510M 65M 446M 13% /boot/firmware
/dev/sda1 ext4 1.8T 808G 932G 47% /mnt/hdd
tmpfs tmpfs 781M 32K 781M 1% /run/user/1000
tmpfs tmpfs 781M 32K 781M 1% /run/user/1001
But on the Core Lightning - there seems to be a problem on the plugins @openoms can you take a look
*** CORE LIGHTNING (MAINNET) SYSTEMD STATUS ***
○ lightningd.service - c-lightning daemon on mainnet
Loaded: loaded (/etc/systemd/system/lightningd.service; enabled; preset: enabled)
Active: inactive (dead) (Result: exit-code) since Sat 2025-01-25 08:07:19 GMT; 2 days ago
Process: 110013 ExecStartPre=/home/admin/config.scripts/cl.check.sh prestart mainnet (code=exited, status=0/SUCCESS)
Process: 110041 ExecStart=/bin/sh -c /usr/local/bin/lightningd --conf=/home/bitcoin/.lightning/config --pid-file=/run/lightningd/lightningd.pid --rpc-file-mode 0660 (code=exited, status=1/FAILURE)
Process: 110047 ExecStartPost=/home/admin/config.scripts/cl.check.sh poststart mainnet (code=exited, status=0/SUCCESS)
Main PID: 110041 (code=exited, status=1/FAILURE)
CPU: 2.979s
Notice: journal has been rotated since unit was started, output may be incomplete.
*** LAST CORE LIGHTNING (MAINNET) INFO LOGS ***
For details also use command --> cllog
sudo tail -n 50 /home/bitcoin/.lightning/bitcoin/cl.log
2025-01-25T08:04:09.652Z INFO lightningd: v24.11
2025-01-25T08:04:11.093Z INFO plugin-clnrest: Killing plugin: disabled itself: No module named 'gunicorn'
2025-01-25T08:04:11.232Z UNUSUAL plugin-bookkeeper: topic 'utxo_deposit' is not a known notification topic
2025-01-25T08:04:11.232Z UNUSUAL plugin-bookkeeper: topic 'utxo_spend' is not a known notification topic
That is a non fatal error with CLN resulting from using the old install script to update.
Reflashing the SDcard with the new release will solve all this so would advise that. Use an Endurance type card preferably 64GB now.
