aws-cli icon indicating copy to clipboard operation
aws-cli copied to clipboard

Support S3 transferring use "ProcessPoolExecutor" with s3tranfer

Open kimsehwan96 opened this issue 6 months ago • 2 comments

Describe the feature

As I know, aws-cli uses "s3tranfer" to use s3 and it uses ThreadPoolExecutor like this https://github.com/boto/s3transfer/blob/da68b50bb5a6b0c342ad0d87f9b1f80ab81dffce/s3transfer/futures.py#L402-L403

In some environment like enough available network bandwidth, enough CPU cores and lots for files to be downloaded, then using ProcessPoolExecutor would be better. And s3transfer has implemented an interface to use ProcessPoolExecutor

https://github.com/boto/s3transfer/blob/develop/s3transfer/processpool.py

So I think addding feature flag for selecting thread or process for using s3 that could be determined by user would be better.

Use Case

If we have to download many files and the environment that using aws-cli has enough resources(CPU, Memory, Network bandwitdh) than we can choice that use more CPUs for boost the S3 throughput

Proposed Solution

No response

Other Information

No response

Acknowledgements

  • [ ] I may be able to implement this feature request
  • [ ] This feature might incur a breaking change

CLI version used

2.15.30

Environment details (OS name and version, etc.)

Amazon Linux 2023

kimsehwan96 avatar Aug 07 '24 03:08 kimsehwan96