TransferManager S3 parallel download poor disk performance and ultimately kills server

[vdemarcus]

downloading with the aws-sdk-java with the following code: final TransferManager xferMgr = TransferManagerBuilder.standard().build(); Download xfer = xferMgr.download("mybucket", "data", new File("/data/data")); xfer.waitForCompletion(); xferMgr.shutdownNow();

where "data" is an S3 file ~ 50GB in 5000 parts with a 5 Gbits network connection to S3

with latest version (1.11.721): works fine on EC2 SSD servers ultimately kills my EC2 d2.4xlarge (HDD storage as a large volume striped over 12 disks) when downloading several of these files.

with version 1.11.457: works fine on EC2 SSD servers works fine my EC2 d2.4xlarge (HDD storage as a large volume striped over 12 disks)

explanation: older version 1.11.457 creates one file per part and then merge the parts to generate the full file new version 1.11.721 creates the final file and each part download does a seek into the right position in the final file and stores it's value I believe this is creating lots of out of order seeks which the hard drives can not keep up with given the size of the final file (50GB) and the number of seeks required (5000 parts)

relevant parts of code: version 1.11.457 -> DownloadPartCallable version 1.11.721 -> DownloadS3ObjectCallable public Long call() throws Exception { RandomAccessFile randomAccessFile = new RandomAccessFile(destinationFile, "rw"); FileChannel channel = randomAccessFile.getChannel(); channel.position(position);

I believe there are no alternatives for TransferManager in the v2 API. Correct me if I am wrong. As a result is that a use-case we could fix in the API v1? (parallel download of large files with lots of parts on a HDD based storage system)

thanks

[zoewangg]

Hi, @vdemarcus I'd like to understand a bit more of your use case. Is there any reason you have so many parts in one file? In our tests, more parts can cause negative performance impacts.

V2 transfer manager is not implemented yet. tracking in https://github.com/aws/aws-sdk-java-v2/issues/37

[vdemarcus]

Hi @zoewangg , the files I am using have been uploaded to s3 by someone else, and I have no control over the number of parts they are using during the upload. It also looks to me like there is no way to change the number of parts when the file is already stored in S3.

A possible solution would be to improve TransferManager to group parts together when downloading (task1: download parts 1-10, etc)

[zoewangg]

I see. Yeah, there's no way to change the number of parts once it's uploaded. We are concentrating efforts on V2 new features at the moment and I don't think we will implement the feature any time soon.

As a workaround, you can try downloading with presignedUrl. For this operation, parallel downloading is implemented using range get instead of multiparts. The default download size is 5MB and you can increase it based on your use case.

https://github.com/aws/aws-sdk-java/blob/ccfd63c097874e3a1e9ffda7bf171769b6bd3b21/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/TransferManager.java#L1207

https://github.com/aws/aws-sdk-java/blob/ccfd63c097874e3a1e9ffda7bf171769b6bd3b21/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/PresignedUrlDownloadConfig.java#L122-L128

[vdemarcus]

Thanks for your suggestion, I'll try it as soon as I can - hopefully I'll be able to get around to it this week

[vdemarcus]

I tried your suggestion using presignedUrl dividing the range in 50 chunks and unfortunately, it doesn't make the download process more stable. The only way I get satisfactory stability is when I use an older version of the library (1.11.457) which does save each individual chunk as a different file, and then proceeds to merge the files. I believe the stability issue is caused by doing: FileChannel channel = randomAccessFile.getChannel(); channel.position(position); on a relatively large file (50GB) with regular HDD (not SSD).

To fix this issue, I believe the library should enforce writing the chunks to the result file in order (always appending) and keep a limited size memory buffer of the chunks not currently written to disk.

I feel it is very disapointing the S3 library coupled with a recent EC2 server are not fully compatible. What should have been a very straight forward process from my side turned into a week long project, trying to get an acceptable level of stability.

I would like to point out the current solution is underwhelming performance wise as writing each of the chunks and then proceeding to write the full file to disk does take indeed twice the disk bandwith and slows down the overall performance of my server.

If you don't fix this in the v1 API, I hope you can at least take this feedback on board for the v2 api.

[mverrilli]

Regarding the change made to use RandomAccessFile, consider the use case where a customer is targeting EFS or a NAS. The performance degradation is significant as compared to the .part file method used previously.

https://github.com/aws/aws-sdk-java/blob/d5b3c6fac47f8273cceed31bd47136c23d308129/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/internal/DownloadPartCallable.java#L47

[github-actions[bot]]

This is a very old issue that is probably not getting as much attention as it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to provide a comment or open a new issue.