benji icon indicating copy to clipboard operation
benji copied to clipboard

Add override-lock to other options

Open adambmedent opened this issue 6 years ago • 8 comments

Benji has been going really well. However we just hit this.

(benji) [root@ceph-benji ~]# /usr/local/benji/bin/benji restore -f --sparse V0000001452 file:/benji-bunker/test.img INFO: $ /usr/local/benji/bin/benji restore -f --sparse V0000001452 file:/benji-bunker/test.img ERROR: AlreadyLocked: Version V0000001452 is already locked.

Anyway we can get --override-locks added to the restore option?

adambmedent avatar Aug 21 '19 15:08 adambmedent

I'm not sure if adding an --override-locks option would be the right approach here. Could you please look into the locks table and send my the corresponding row data? That way we could figure out what operation acquired the lock. If the operation holding the lock is/was a backup then I'm not sure if a restore should be allowed at all as the resulting image would be incomplete. Or maybe it would be better to don't acquire the lock on restore and just output a warning when the version is locked.

(If you really need a restore of this version you can manually delete the corresponding row from the locks table. After collecting the info mentioned above of course...)

elemental-lf avatar Aug 22 '19 10:08 elemental-lf

I actually figure out how to delete the lock yesterday so I won't be able to provide that information.

However I can tell you that we hit the lock issue after ctrl+c out of a restore process.

adambmedent avatar Aug 22 '19 10:08 adambmedent

Okay, so it probably is a left-over of the interrupted restore process (seems that lock clean-up isn't working in all cases). I will have to think about this but I am currently leaning in the direction of not locking the version at all on restore and just issuing a warning. EDIT: Another option would be to add a general "break locks" command.

elemental-lf avatar Aug 22 '19 12:08 elemental-lf

I have done a bit of testing and this is what I see.

If a ctrl+c out of a restore and let the cleanup run, all is well. However if I ctrl+c the restore and hit ctrl+c a couple more times, it also seems to kill that cleanup process. Not sure if that is something you can address or not.

adambmedent avatar Aug 22 '19 17:08 adambmedent

While it would be possible to try to catch or ignore these additional SIGINTs I don't think that it is really sustainable long-term. The locking on versions is mainly done to prevent removal of the version while on operation on it is in progress. I'm thinking about replacing the locking with a lease based system. That way things would clean themselves up automatically as soon as the lease expires.

elemental-lf avatar Aug 23 '19 09:08 elemental-lf

I like the sounds of that, seems like it would solve multiple issues.

adambmedent avatar Aug 23 '19 13:08 adambmedent

Great! I just came here to report the same issue. This sounds like a good aproach.

But possibly an additional "force-unlock" command would be nice, depending on the intended lease-duration.

wech71 avatar Jan 21 '20 09:01 wech71

My current plan is to go for a short lease duration (minutes) and renew the lease regularly while an operation is in progress.

elemental-lf avatar Jan 21 '20 12:01 elemental-lf