orb-software
orb-software copied to clipboard
hil: Use multi-part download for hil
AWS rust sdk is slow compared to cli, multipart download can speedup the process significantly, see https://github.com/awslabs/aws-sdk-rust/discussions/1024.
The logic for the change is mostly adapted from: https://github.com/smithy-lang/smithy-rs/blob/main/aws/sdk/benchmarks/s3-throughput/benchmark/src/multipart_get.rs.
For me the speedup was around 2x in Munich(It should be different for different latencies).
To be noted, that requests to s3 do cost something, but as I understood it's insignificant amount: https://aws.amazon.com/s3/pricing/?p=pm&c=s3&z=4. Lmk if I missed anything.
0.0004 $ per 1000 requests and with multi-part download we have ~40 requests per GB.