cacache-rs icon indicating copy to clipboard operation
cacache-rs copied to clipboard

feature request: tokio::io::AsyncSeek support for Reader

Open rbtcollins opened this issue 2 years ago • 5 comments

Use case: I have zip files being added to a cacache instance, and zip is a non-streamable format - the index of a zip is at the end, so it is somewhat inefficient to read everything, then scan the index, then reopen and re-read to extract the actual content (using async_zip)

rbtcollins avatar Oct 03 '23 19:10 rbtcollins

sure, this sounds good.

Ideally, we'll also have the async-std version of this done at the same time, though.

zkat avatar Oct 04 '23 18:10 zkat

What do you think the API would look like @zkat ? The Reader in cacache also performs integrity checks, if we could seek in the file this becomes much harder.

baszalmstra avatar Nov 30 '23 15:11 baszalmstra

yeah I have no idea how that would work tbh

zkat avatar Nov 30 '23 19:11 zkat

I think you could read through to do the verification, then let the caller use seek to read the bits they need; the main point is to not have to buffer the entire file in RAM or make a temp copy of it, when using a non-streamable layer on top

rbtcollins avatar Apr 15 '24 20:04 rbtcollins

Yeah. And we can have the unchecked and checked versions, in case you don't want to read the whole thing because, e.g. the file is enormous.

Ok.

I'll take a PR for this if anyone's interested, but it would need to be implemented for tokio, async-std, and sync. I don't want feature drift between the backends.

zkat avatar Apr 15 '24 21:04 zkat