mcap icon indicating copy to clipboard operation
mcap copied to clipboard

CLI: fast summary section rewrites

Open wkalt opened this issue 1 year ago • 3 comments

Currently the add attachment and add metadata subcommands rewrite the entire file, which takes time proportional to the size of the file. It would be way faster if we only rewrote the summary section. For this we need some tricky code to bump all the appropriate summary section indexes, depending on where the summary offsets for the metadata/attachment sections reside, but it should totally be possible.

One thing to be aware of - we will lose crash safety, vs the current tmpfile + move approach. Options to mitigate seem like a) copy the original file to a "original file" location, which has the downside of still being proportional to file size, or b) pairing this with a robust "reindex" subcommand that will regenerate a valid summary section from the presumed-valid original data section (we should not need to touch the data section). Option b) seems best IMO, to make the average case very fast.

wkalt avatar Oct 25 '22 02:10 wkalt