start-os
start-os copied to clipboard
Start9Labs
Add indicator to backup page informing user how much data is being backed up
so they know whether it's frozen or just backing up a ton of data
Reported by: @chrisguida
Add PackageProcedure to backup api such that a service can estimate how much data to back up. Ideally, allow it to mount read-only to the backup target so it can exclude data that has already been backed up (for incremental backups).
Add PackageProcedure to backup api such that a service can estimate how much data to back up. Ideally, allow it to mount read-only to the backup target so it can exclude data that has already been backed up (for incremental backups).
I find it difficult to believe anyone will use this feature. Perhaps we ought to just inform the user a moving window totaling how much data has been written to the backup drive in the last N minutes. This will make it clear that shit is still going but prevents us or the package developer from having to predict these things at the outset.
Some kind of display saying "you are about to backup X GB of data" seems eminently useful, and not all that difficult, especially in the case where we don't have to calculate the diff
No I am not challenging that it is useful to the user, I'm challenging that devs will opt to do the work of computing how much data will need to be backed up in the presence of diffing. Saying "[this is] not all that difficult" misunderstands the problem. If we implement an available package procedure to compute this, my hypothesis is that external devs will opt to implement it correctly.
Right, i mentioned that calculating the diff would be the more difficult case.
In the base case, it seems to me like this could be implemented at the OS level, just take du of the whole datadir and subtract the size of whatever's in .backupignore.
I suppose where this gets complicated is if the package has some kind of custom backup mechanism that doesn't use compat or duplicity
That's my point exactly. While it is unlikely that any backup action will expand the volume data during the backup process, it is technically possible to do so. Due to Turing Completeness, this can only be computed by the package itself, the OS CANNOT do it on its own. The heuristic above may serve as an upper bound estimate but that's all it would be.
For all our current services, this would be easy.
I would definitely implement this for all our current services, I assume it will be pretty much all boilerplate
In the base case, it seems to me like this could be implemented at the OS level, just take du of the whole datadir and subtract the size of whatever's in .backupignore.
^
@dr-bonez and I discussed that, in the absence of a package procedure, the OS would naively measure the entire volume and report that as the estimate along with a warning that it might not be accurate.
No I am not challenging that it is useful to the user, I'm challenging that devs will opt to do the work of computing how much data will need to be backed up in the presence of diffing. Saying "[this is] not all that difficult" misunderstands the problem. If we implement an available package procedure to compute this, my hypothesis is that external devs will opt to implement it correctly.
If we offer easy, off the shelf ways to do this, then I don't think it will be an issue. So as long as we can figure out a way to do this in duplicity, I think it will be used, as long as it can be a quick shorthand to invoke that procedure.