pget icon indicating copy to clipboard operation
pget copied to clipboard

Enhancement: support larger file download

Open anotherjesse opened this issue 2 years ago • 3 comments

@daanelson has shared that https://storage.googleapis.com/replicate-weights/llama-13b-fp16.tensors which is ~24GB.

To compare, gcloud can download this in parallel between 1-2 GBps

anotherjesse avatar Jun 27 '23 18:06 anotherjesse

I just have an anecdotal sample size, but I've found that pget works as-is for that model when run on a A100 instance.

Download time with pget was between 21-24 seconds. I tried tweaking -c and found that -c 10 to -c 12 seemed to slightly improve on speed obtained with the default.

Tests with gcloud yielded downloads between 16-24 seconds (with download speeds ranging from 1.1-1.7 GBS).

joehoover avatar Jun 28 '23 03:06 joehoover

Potentially it makes sense to also compare to available ram, if one cannot buffer the whole file into memory use scratchspace and bind files together after.

tempusfrangit avatar Aug 04 '23 19:08 tempusfrangit

Partially covered by #177

tempusfrangit avatar Mar 07 '24 11:03 tempusfrangit