extend the backup mechanism during updates

[muued]

make it optional (enabled by default). only include relevant application parts. exclude docs, consumption, media. compress the backup.

[muued]

As far as I understand it, this introduces a dependency to the community.general collection of ansible if using ansible version 3. Additonally, the target system needs to have gzip installed for this to work. I guess debian and ubuntu both have that capability out of the box. On a fresh test machine, ubuntu came with gzip installed (or it was installed as part of the paperless-ng installtion).

[muued]

To restore a backup created during an update, one would use tar -xf /opt/paperless-ng-12341212T12341212.tgz -C / with the dummy timestamp replaced with the actual timestamp, of course.

[jonaswinkler]

I can't really comment on any of that, I don't use ansible.

[anokfireball]

this introduces a dependency to the community.general collection

That might already be the case, I'm not sure though. Please include this information in the Dependencies section of the ansible README.md (https://github.com/jonaswinkler/paperless-ng/tree/master/ansible#dependencies).

the target system needs to have gzip installed for this to work

Please include this dependency in the install base dependencies step. Even though Debian and Ubuntu (currently) ship with gzip by default, making the requirements explicit will be useful when adding different operating systems.

To restore a backup created during an update, one would use tar -xf /opt/paperless-ng-12341212T12341212.tgz -C /

Please also include this information somewhere in the code (either the README.md, the Updating Paperless documentation or in the ansible source directly). It will be easier to find it there rather than in the comments of the PR that introduced it.

[anokfireball]

On a sidenote, have you tried restoring a backup created this way?

[muued]

On a sidenote, have you tried restoring a backup created this way?

Yes, I successfully restored an old version this way, however it comes with a series of caveats. Of course, the paperless services should be stopped before doing anything. Extracting the archive without deleting anything beforehand works, but may have unintended side effects, as stuff that was added might be lying around where it shouldn't. Im my simple test, all was good. But... If you "prepare" the root directory by deleting anything inside the paperlessng root directory aside from the consumption and media folders (in case those reside within that folder), restoring the backup may lead to permission problems, since the .venv, data, scripts, src, and static folders themselves are not part of the backup, only their contents (which came as a little surprise for me). They are created on the fly, but by the user performing the restore and get default permissions, which may not suffice for the application to work. This fact may hint at simplifying the included files/folders to https://github.com/jonaswinkler/paperless-ng/pull/1058#discussion_r638250910 and move the question of what's important for the application to run to the actual installation phase. However, it would still be required to restore the ownership and permissions of any folder created on the fly (which was previously manually deleted by the user), in case they are not contained in a folder explicitly included in the backup paths. This will include the paperless root directory if deleted and (if moved outside of the root folder and deleted) the data, static, and virtualenv folders.

The command I gave above only makes sense when run as root, otherwise you have to add the --same-owner --preserve-permissions flags.

[muued]

To restore a backup created during an update, one would use tar -xf /opt/paperless-ng-12341212T12341212.tgz -C /

Please also include this information somewhere in the code (either the README.md, the Updating Paperless documentation or in the ansible source directly). It will be easier to find it there rather than in the comments of the PR that introduced it.

I will add that information as soon as the general approach looks OK to you. The documentation seems to be the right place to mention this in my opinion. The question is, whether to put it under Updating Paperless oder Downgrading Paperless. It should also mention the potential necessity of a manual database migration, which is currently the only topic under "Downgrading Paperless".

[jonaswinkler]

@C0nsultant I'm not really following the ansible discussion; would you mind notifying me when this is good for merging?

[muued]

In my opinion, taking into account all potential configuration options would require a considerable amount of code that would need to be maintained. Thus, I propose to give instructions on how to restore a backup using the default values and assume that users are able to adjust the paths in the code snippet in case they made configuration adjustments. However, I tried to make it as general as I could, so that one just has to replace the string /opt/paperless-ng with whatever was used instead for paperlessng_directory as well as the system user and group.

# stop the paperless services
sudo systemctl stop paperless-{webserver,scheduler,consumer}

# move the broken installation to a temporary directory
# in case paperlessng_data_dir, paperlessng_staticdir, paperlessng_virtualenv resided outside of paperlessng_directory, move them away on your own
sudo mv /opt/paperless-ng /opt/paperless-ng-broken

# restore a backup
sudo tar -xf /opt/paperless-ng-20200202101010.tgz -C /

# fix folder permissiongs
# in case paperlessng_data_dir, paperlessng_staticdir, paperlessng_virtualenv resided outside of paperlessng_directory, fix permissions on your own
sudo find /opt/paperless-ng -maxdepth 1 -type d -exec chown paperlessng:paperlessng {} \; -exec chmod 750 {} \;

# move the parts not contained in the backup into the restored installation
# ignore any errors in case you had configured the corresponding folders to reside outside of paperlessng_directory
sudo mv /opt/paperless-ng-broken/{consumption,media} /opt/paperless-ng/

# potentially downgrade the database, see https://paperless-ng.readthedocs.io/en/latest/administration.html#downgrading-paperless
sudo systemctl start paperless-{webserver,scheduler,consumer}

# make sure everything is working again
# then delete the remains of the broken installation
sudo rm -r /opt/paperless-ng-broken

[muued]

@C0nsultant it would be great if you could have a look at the proposed changes and tell me/us if there's anything that should be improved.