Backend repository's been locked after crashing job

[shudipta]

If a job crashes while backing up data, the sub-sequent jobs also crash saying,

I0718 12:03:56.470928       1 commands.go:247] Checking integrity of repository
[golang-sh]$ /bin/restic_0.9.5 check --cache-dir /tmp/restic-cache
Fatal: unable to create lock in backend: repository is already locked by PID 99 on stash-backup-pxc-cluster-backup-1563450601-qtc7j by root (UID 0, GID 0)
lock was created at 2019-07-18 11:59:11 (4m47.890669629s ago)
storage ID 4400ff6b

Need to release the lock.

[lorenzomorandini]

Hi, having the same problem here. What is the correct procedure to manually release the lock?

[hossainemruz]

@lorenzomorandini try this: https://stash.run/docs/v0.9.0-rc.2/guides/latest/cli/cli/

[lorenzomorandini]

@hossainemruz Thank you but I'm still on 0.8.3. Is there a way to do it? Backups of that deployment have been blocked for a few days now.

[lorenzomorandini]

Solved by manually exec into stash container and run restic unlock -r <repository_mount_path>

[LeoQuote]

@lorenzomorandini hello, what is the repository_mount_path ? I tried /tmp/restic_cache/ and mountPath of my pvc , but no work

[LeoQuote]

I think I found it anyway, I deleted lock files from s3 storage directly.

[aarononeal]

It is not uncommon for repositories to get locked due to a crash, and manually repairing these is tedious.

Since by default unlock only removes stale locks, could we have an option to run this as a pre-backup step?

I've had to add unlock manually to all my jobs as a pre-backup hook, but this has the following drawbacks:

1. It uses up the one and only one pre-backup hook. It's a shame there is not an array of hooks as multi-steps from different containers is useful.
2. I can only add the hook after the first backup because the repo needs to exist else the command fails and prevents first backup.
3. This depends on having manually configured generated cron jobs with concurrencyPolicy: Forbid to prevent jobs from overlapping, since stash lacks this as well. And since PVC backups run as init containers, which that doesn't work with, I had to convert all my PVC backups to busybox with sidecars.

[hossainemruz]

Thank you @aarononeal for your feedback. We will work on it.

[elcomtik]

I experienced a locked repository recently. Is there any progress on a systematic solution for this?