borgbackup
directory content is only implicitly known
Adding suggested labels based on user approval: enhancement for issue #32, enhancement for issue #191, enhancement for issue #641, enhancement for issue #939. No labels removed.
Can we efficiently access random items in metadata[b'items']? The code accessing that so far is strictly linear access.
If we could, a directory tree index could be added separately, only containing indexes into the items. This could speed up all content-reading actions significantly if only a subset is accessed.
Edit: Ah, no, msgpack only has a streaming unpacker it seems. But maybe we can seek before feeding data to unpacker and this way emulate random access?
basically to make this safe, there is need for dependency tracking, so metadata of a directory will only get written after its content is done
this could be tracked as tree of contents that constructs as unpacking is done, and deconstructs as items are written,
every time a treee turns empty, we can write directory metadata
the pipeline can be created in a way that a tree cannot turn empty unless all its content is processed
That would make it unnecessary to make another pass over all dirs in borg extract, but doesn't speed up "random access" to directories / files for listing/extracting subtrees.
Concrete example: listing all files in /home takes a minute or so, listing a subtree with a few dozen files still takes ~20 s (because iter_items / unpack_many needs to read, unpack, decode, filter all items)
well, a backup tool has to make some trade-offs, - since the metadata is supposed to be sorted, i believe that binary search is reasonably fast for that kind of access
If I understand things correctly accessing items non-linearly is not trivial to do.
(I'm not saying that this is an actual problem that needs solving, I think it's "fast enough", at least for me)