Backup is out of date, we cannot continue safely. Emergency shutdown.

[manelio]

trafficstars

I'm using lightningd via raspiblitz.

After restarting after a power failure lightningd is not running anymore.

When I try to start the service manually I receive an error message Backup is out of date, we cannot continue safely. Emergency shutdown.:

sudo -u bitcoin /usr/local/bin/lightningd --conf=/home/bitcoin/.lightning/config
Plugin '/home/bitcoin/cl-plugins-enabled/backup.py' returned an invalid response to the db_write hook: 

{"jsonrpc": "2.0", "id": 11, "error": {"code": -32600, "message": "Error while processing db_write: 'Process' object has no attribute 'parents'", "traceback": "Traceback (most recent call last):\n  File \"/home/bitcoin/.local/lib/python3.7/site-packages/pyln/client/plugin.py\", line 621, in _dispatch_request\n    result = self._exec_func(method.func, request)\n  File \"/home/bitcoin/.local/lib/python3.7/site-packages/pyln/client/plugin.py\", line 606, in _exec_func\n    return func(*ba.args, **ba.kwargs)\n  File \"/home/bitcoin/cl-plugins-enabled/backup.py\", line 63, in on_db_write\n    assert(check_first_write(plugin, change.version))\n  File \"/home/bitcoin/cl-plugins-enabled/backup.py\", line 56, in check_first_write\n    kill(\"Backup is out of date, we cannot continue safely. Emergency shutdown.\")\n  File \"/home/bitcoin/cl-plugins-enabled/backup.py\", line 110, in kill\n    procs = [p for p in psutil.Process(os.getpid()).parents()]\nAttributeError: 'Process' object has no attribute 'parents'\n"}}

Is there any way to fix this problem?

[cdecker]

This happens when the node was started without the backup plugin, i.e., it's missing a couple of changes that were applied to the original DB. If the main DB is working you can just re-initialize the backup file (delete then backup-cli init) and it'll work again. We can't do this automatically since it could have been caused by a faulty restore and we don't want to delete databases willy nilly.

[PestToast]

@manelio did that resolve your issue?

[ghost]

I have the same issue. What I did is to backup the out of date/corrupted(?) DB(.sqlite3 and .sqlite3-journal) on the lightning dir and external backup file, just to make sure it is safe. And then do what @cdecker told, to remove the external backup file and re-init the backup plugin. I guess it fixed the problem.

[http402]

i have the same issue after raspiblitz crashed, maybe due to overheating? was running a lot of channels and two nodes...

could somebody walk me through the steps of recovering? i'm not sure what data to backup and how, and what needs to be removed.

[http402]

This happens when the node was started without the backup plugin, i.e., it's missing a couple of changes that were applied to the original DB. If the main DB is working you can just re-initialize the backup file (delete then backup-cli init) and it'll work again. We can't do this automatically since it could have been caused by a faulty restore and we don't want to delete databases willy nilly.

how do i know if the main DB is working?

/home/bitcoin/.lightning/bitcoin/cl.log outputs:

2022-03-26T20:37:32.997Z INFO    plugin-backup.py: root        : Comparing backup version 1329293 versus first write version 1329295
2022-03-26T20:37:32.998Z INFO    plugin-backup.py: Comparing backup version 1329293 versus first write version 1329295
2022-03-26T20:37:32.998Z INFO    plugin-backup.py: Backup is out of date, we cannot continue safely. Emergency shutdown.
2022-03-26T20:37:34.001Z INFO    plugin-backup.py: Killing process lightningd (236288)

[http402]

fixed my node by running

config.scripts/cl-plugin.backup.sh on

following this: https://github.com/rootzoll/raspiblitz/issues/3015