Failing database backups should not lead into ioBroker crash

[thohoj]

Hi,

I wanted to redirect the backup_objects directory from IOBroker installation to a network drive (to reduce the number of writes to the SD card of my Raspberry Pi). I have done this by symlinking into a NFS directory.

If there is a problem with NFS, network or server the ioBroker start fails.

$IOB version
4.0.21
$ iob start
$ iob status
Uncaught Rejection: Error: ENOENT: no such file or directory, mkdir '/opt/iobroker/iobroker-data/backup-objects'
    at Object.mkdirSync (fs.js:1013:3)
    at Object.module.exports.makeDirSync (/opt/iobroker/node_modules/@iobroker/db-base/node_modules/fs-extra/lib/mkdirs/make-dir.js:23:13)
    at ObjectsInMemoryServer.initBackupDir (/opt/iobroker/node_modules/@iobroker/db-base/lib/inMemFileDB.js:205:12)
    at new InMemoryFileDB (/opt/iobroker/node_modules/@iobroker/db-base/lib/inMemFileDB.js:85:18)
    at new ObjectsInMemoryFileDB (/opt/iobroker/node_modules/@iobroker/db-objects-file/lib/objects/objectsInMemFileDB.js:34:9)
    at new ObjectsInMemoryJsonlDB (/opt/iobroker/node_modules/@iobroker/db-objects-jsonl/lib/objects/objectsInMemJsonlDB.js:58:9)
    at new ObjectsInMemoryServer (/opt/iobroker/node_modules/@iobroker/db-objects-jsonl/lib/objects/objectsInMemServerRedis.js:57:9)
    at new ObjectsInMemoryServerClass (/opt/iobroker/node_modules/@iobroker/db-objects-jsonl/lib/objects/objectsInMemServerClass.js:34:30)
    at Timeout._onTimeout (/opt/iobroker/node_modules/iobroker.js-controller/lib/setup.js:3123:27)

IMHO this exception should be caught and ioBroker should continue as normal (so my wife can get back into the house without a locksmith).

[Apollon77]

So you mean when an error occurs that prevent us from doing needed backups that often helped a lot the controller should start?? Honestly ... maybe a bad idea because the user wil most likely never see logs about it or such - so maybe you should rethink your strategy? ;-)

ALso The backup is done not that often ... the normal DB files are written much more often, so I do not think that this brings that much benefit, or?!

Yes we should catch the error, here I totally agree ;-) But what to do when it happens is an interesting question ...

[thohoj]

backup_objects is written every 2 hours (multiple files) to. When a backup fails or if the backup directory cannot be created the normal ioBroker tasks should continue. The admin user should get an alert email in case of a failure. I think this is a standard procedure in the automation sector.

[Apollon77]

Right now iobroker can not send an emails by default. So error notifies are up to the user to setup. And also if we add an admin notification users only see it when the open admin. It is true that iobroker should get standard way to notify a user and this is also on the idea list.

But back to the topic in general - first of all we will discuss internally how to react in such an error case and will consider your opinion.

And my question is still why you think it is a good idea to combine an unreliable Storage deep into the „internal iobroker system directory“ because honestly this is causing the issue the very first time of all current 60k installations ;-)

[thohoj]

You are right, basically it is not a good idea. But in backup_objects are only backups of databases, made every 2 hours (why so often? What about a better retention strategy, ...). This is wearing my SD card and I make backups for my own (backitup and others).

So in my case (with own backups): would it be OK to make the backup_objects readonly and prevent the creation of backups (in backup_objects)? Or is it needed for other tasks I do not know?

or better: expose settings so I can change the backup behavior in the GUI.

[Apollon77]

made every 2 hours (why so often? What about a better retention strategy, ...

It is all configurable in the backup sections of "iobroker-data/iobroker.json" (or in admin). In fact the intervals are coming from the experience of user-needs from the past years. They are also cleaned up ... I do not knwo what you mean with "better retention strategy".

Because these backup files are packed they are not that big usually - and the normal DB and logs produce a lot more "wearing" than these backup files, or?!

So in my case (with own backups): would it be OK to make the backup_objects readonly and prevent the creation of backups (in backup_objects)? Or is it needed for other tasks I do not know?

We are still in the internal clarification, but because these backup files are sometimes very important for users - especially when their SD card breaks - we do not want to "hide" tat these backups can not be created, so we still favor "not starting iobroker if these backups can not be created.

or better: expose settings so I can change the backup behavior in the GUI.

Already there. Use the latest available Admin version !!! Enable expert mode, Tab "Hosts" and klick the wrench, then go to states/objects and change the backup frequency or such.

[thohoj]

Found it ("on-the-fly backup")! Thank you.

With "No on-the-fly backup" I can completly turn off the backup, perfect.

And one can select a special alternative directory to store the backups, so there would be no need of replacing the internal ioBroker backup_objects with a symbolic link.

I do not knwo what you mean with "better retention strategy"

An example for a better retention strategy would be:

• Make backups every 2 Hours. Delete the oldest, if a maximum file count of 24 is reached
• every day: copy/keep the latest "2 hour" backup. Delete the oldest daily backup, if a maximum file count of 12 is reached.
• every week: copy/keep the latest "daily" backup. Delete the oldest weekly backup, if a maximum file count of 4 is reached.
• every month: copy/keep the latest "weekly" backup. Delete the oldest monthly backup, if a maximum file count of 12 is reached.
• every year: ...

If something is wrong with your database/objects/whatever, although the backup is working, it would backup the defect data set. And with the current retention strategy the last "good" backup will be deleted after 48h / 24 backups. With the example above

• a few "old" backups will be kept for a longer time
• more "young" backups will be kept for a shorter time

I am using a similar strategy for backing up some data on my server using rsnapshot. It uses hardlinks for efficient storage.

-- Thank you for your support. Great work.

[thohoj]

If I check the "No on-the-fly backup" checkboxes (objects and states) after a restart of the server the checkboxes are unchecked again.

[thohoj]

Disabling on-the-fly-backups by manually editing iobroker-data/iobroker.json works.

[Apollon77]

An example for a better retention strategy would be:

Really? This is about the DB files itself. Who needs a backup from a year for it's smarthome? The reality is that you need the last reent working backup ... Anything else is too old ... So yes your thought might be rigth for te complete backups ...

If I check the "No on-the-fly backup" checkboxes (objects and states) after a restart of the server the checkboxes are unchecked again.

Then please open an admin issue. Thank you!

[thohoj]

This is only an example. After one week backing up a more and more corrupted database you were happy to have a two week old good copy. But with the solution above that does no longer concern me.

[Apollon77]

The automatic db backups DO NOT replace real Backups!! Relying on these db backups is a bad idea. Please reply on real backups

[thohoj]

You're right there with me. That's why I disabled the ioBroker backups. I just didn't know how before.

[foxriver76]

We can catch and log a more user friendly error but as discussed we should not start.