pkgx icon indicating copy to clipboard operation
pkgx copied to clipboard

extract the tarball while downloading

Open mxcl opened this issue 3 years ago • 8 comments

pipe the data to an instance of tar, extract as we download.

mxcl avatar Dec 02 '22 16:12 mxcl

failure cases:

  • download doesn't complete
  • sig/checksum verification doesn't match

those seem like risks that will be harder to mitigate. thoughts?

jhheider avatar Dec 02 '22 18:12 jhheider

I feel like pipelining of downloads (such that all download+extract steps occur in parallel) is probably "good enough", and in general most tarballs are not as big as e.g. LLVM.

jonchang avatar Dec 02 '22 18:12 jonchang

@jonchang do you mean running downloads of multiple packages in parallel but executing download/verify/unpack in series per package, or do you mean download+unpack in a stream, and verify afterwards for any given package?

jhheider avatar Dec 02 '22 19:12 jhheider

download doesn't complete

delete extraction is sufficient I assume

sig/checksum verification doesn't match

same…?

unless I'm missing something.

mxcl avatar Dec 02 '22 20:12 mxcl

my goals here are for us to be suppppper fast

parallel downloads/verify/extract being a next step, but I'm worried about the UI for that so plan to do it myself. Since I wrote the current TUI for it.

mxcl avatar Dec 02 '22 20:12 mxcl

I suppose I'm worrying about the case where we're replacing existing/working bins, but that's mostly only on build, so deletion of the target directory might be sufficient.

jhheider avatar Dec 02 '22 20:12 jhheider

replacing existing/working bins

tea can't do this currently. build is certainly different and I'd like us to work better here but currently don't have any great solutions (bar two tea prefixes, which is maybe the best bet, but a little tricky to engineer).

mxcl avatar Dec 02 '22 20:12 mxcl

also we should be able to resume downloads that fail eventually. Not too hard in this modern HTTP world. Currently we cannot and in fact “trying again” fails and deletes the tarball.

mxcl avatar Dec 02 '22 20:12 mxcl

This in 0.25

mxcl avatar Mar 15 '23 14:03 mxcl