orb-software icon indicating copy to clipboard operation
orb-software copied to clipboard

hil: Use multi-part download for hil

Open AlexKaravaev opened this issue 1 year ago • 0 comments

AWS rust sdk is slow compared to cli, multipart download can speedup the process significantly, see https://github.com/awslabs/aws-sdk-rust/discussions/1024.

The logic for the change is mostly adapted from: https://github.com/smithy-lang/smithy-rs/blob/main/aws/sdk/benchmarks/s3-throughput/benchmark/src/multipart_get.rs.

For me the speedup was around 2x in Munich(It should be different for different latencies).

To be noted, that requests to s3 do cost something, but as I understood it's insignificant amount: https://aws.amazon.com/s3/pricing/?p=pm&c=s3&z=4. Lmk if I missed anything.

0.0004 $ per 1000 requests and with multi-part download we have ~40 requests per GB.

AlexKaravaev avatar Oct 09 '24 09:10 AlexKaravaev