aws
S3 Transfer Manager with the AWS CRT-based S3 client is very slow compare to SDK 1.11.49
Describe the bug
Download a file of 3.7 GB Transfer manager SDK 1.11.49 -> Completes in 2 minutes 27 seconds AWS CRT-based S3 client -> Completes in 4 minutes 46 seconds
Expected Behavior
AWS CRT-based S3 client is faster than Transfer manager SDK 1.11.49
Current Behavior
AWS CRT-based S3 client is faster than Transfer manager SDK 1.11.49
Reproduction Steps
S3CrtAsyncClientBuilder builder = S3AsyncClient.crtBuilder().credentialsProvider(ProfileCredentialsProvider.create("XYZ")).region(Region.US_WEST_2);
try (S3TransferManager transferManager = S3TransferManager.builder().s3Client(builder.build()).build()) {
final DownloadFileRequest o = DownloadFileRequest.builder()
.getObjectRequest(req -> req.bucket(bucketName).key(objectName))
.destination(tempFile.getAbsoluteFile())
.addTransferListener(LoggingTransferListener.create())
.build();
FileDownload fileDownload = transferManager.downloadFile(o);
fileDownload.completionFuture().join();
}
Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
2.22.0
JDK version used
Java 8
Operating System and version
Ubuntu Linux
@mailsanchu can you share your v1 TransferManager code sample too, including the S3 client construction? We can probably provide some guidelines on how to tune the v2 TransferManager to improve performance.
Also, what is the specs of the machine/environment you're running the samples? The tuning will depend on the environment specs.
@debora-ito
TransferManagerBuilder transferManagerBuilder = TransferManagerBuilder.standard().withS3Client(s3).withExecutorFactory(() -> Executors.newFixedThreadPool(24)).withMultipartCopyPartSize((long) (25 * 1024 * 1025));
this.s3TransferManager = transferManagerBuilder.build();
Download download = s3TransferManager.download(bucketName, objectName, tempFile);
download.addProgressListener(new MyCustomProgressListener(download.getObjectMetadata().getContentLength(), fileLocation));
download.waitForCompletion();
Specs of the machine model name : 11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz 64 GB Memory
One thing noticed is v1 creates smaller file and merge at the end where as v2 updates the same file
*-cpu
description: CPU
product: 11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz
vendor: Intel Corp.
physical id: 400
bus info: cpu@0
version: 11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz
slot: U3E1
size: 958MHz
capacity: 4900MHz
width: 64 bits
clock: 100MHz
capabilities: lm fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l2 invpcid_single cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid movdiri movdir64b fsrm avx512_vp2intersect md_clear flush_l1d arch_capabilities cpufreq
configuration: cores=8 enabledcores=8 threads=16
TransferManagerBuilder transferManagerBuilder = TransferManagerBuilder.standard().withS3Client(s3).withExecutorFactory(() -> Executors.newFixedThreadPool(24)).withMultipartCopyPartSize((long) (25 * 1024 * 1025));
Can you share the code showing how s3 is being created?
AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard().withRegion(awsS3Region);
builder.withCredentials(new ProfileCredentialsProvider(edgeConfiguration.getLocalSamlAwsProfile()));
if (maxErrorRetry > 0) {
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setRetryPolicy(PredefinedRetryPolicies.getDefaultRetryPolicyWithCustomMaxRetries(maxErrorRetry));
amazonS3ClientBuilder.setClientConfiguration(clientConfiguration);
}
amazonS3 = amazonS3ClientBuilder.build();
Thank you for answering all the questions so far.
Here's one more: what's the network bandwidth in your tests, in that machine?