wyng-backup icon indicating copy to clipboard operation
wyng-backup copied to clipboard

Grouping updates together into single transactions

Open tasket opened this issue 9 months ago • 0 comments

Grouping multiple updates from send (for example) into a single transaction may have advantages for both speed and integrity.

Ideas:

  • Add metadata .tmp updates to a queue in the ArchiveSet, instead of going through the normal save-and-rename chain.
  • Use a global transaction id and append it to the name of each metadata file (instead of using '.tmp' suffix).
  • Do all update_delta_digest() in one batch (for incrementals) and then group the zero-change vols together during send; then process the resulting metadata update queue. This saves a lot of latency overhead when --send-unchanged is used.
  • Possibly use this to simplify recovery from in_process interruption.
  • Possibly re-use tar stream process and start/manage it in monitor_send() instead.

tasket avatar May 20 '24 19:05 tasket