aws-cli
aws-cli copied to clipboard
Support S3 transferring use "ProcessPoolExecutor" with s3tranfer
Describe the feature
As I know, aws-cli uses "s3tranfer" to use s3 and it uses ThreadPoolExecutor
like this
https://github.com/boto/s3transfer/blob/da68b50bb5a6b0c342ad0d87f9b1f80ab81dffce/s3transfer/futures.py#L402-L403
In some environment like enough available network bandwidth, enough CPU cores and lots for files to be downloaded, then using ProcessPoolExecutor
would be better.
And s3transfer
has implemented an interface to use ProcessPoolExecutor
https://github.com/boto/s3transfer/blob/develop/s3transfer/processpool.py
So I think addding feature flag for selecting thread
or process
for using s3 that could be determined by user would be better.
Use Case
If we have to download many files and the environment that using aws-cli has enough resources(CPU, Memory, Network bandwitdh) than we can choice that use more CPUs for boost the S3 throughput
Proposed Solution
No response
Other Information
No response
Acknowledgements
- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
CLI version used
2.15.30
Environment details (OS name and version, etc.)
Amazon Linux 2023