aws
S3 FileDownload fails with file lock exception : FileLockException
Upcoming End-of-Support
- [x] I acknowledge the upcoming end-of-support for AWS SDK for Java v1 was announced, and migration to AWS SDK for Java v2 is recommended.
Describe the bug
Problem: We are encountering FileLockException errors when 42 applications attempt to download files in parallel from S3 for processing.
Error Details:
com.amazonaws.services.s3.transfer.exception.FileLockException: Fail to lock L9_NS_S3_TO_xxxxxxx_011-xxxxxx-test-xxxxx-3-buffer/xxxxx.00.xxxxxx_sxxxx.parquet
at com.amazonaws.services.s3.transfer.DownloadCallable.truncateDestinationFileIfNecessary(DownloadCallable.java:200) ~\[xxxxxx:?\]
at com.amazonaws.services.s3.transfer.DownloadCallable.downloadInParallel(DownloadCallable.java:139) ~\[xxxxxxxx:?\]
at com.amazonaws.services.s3.transfer.DownloadCallable.downloadInParallel(DownloadCallable.java:99) ~\[xxxxx:?\]
at com.amazonaws.services.s3.transfer.internal.AbstractDownloadCallable.call(AbstractDownloadCallable.java:100) ~\[xxxxx:?\]
at com.amazonaws.services.s3.transfer.internal.AbstractDownloadCallable.call(AbstractDownloadCallable.java:40) ~\[xxxxxx:?\]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~\[?:?\]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~\[?:?\]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~\[?:?\]
at java.lang.Thread.run(Thread.java:834) ~\[?:?\]
Context:
- Each of the 42 applications is designed to process unique files, meaning there should be no contention for the same file download to the same system file location.
- The issue is intermittent; after an application retries multiple times on a specific file, it eventually moves on to a different file, only to encounter the FileLockException again on that new file.
- We are always performing delete all downloaded files including File Lock file as well, before starting to download files again.
Java Code For reference
File pathToDownload = new File(bufferedDirectoryName+filenameWithoutPrefix);
transferManager.download(bucketName, filename, pathToDownload).waitForCompletion();
boolean success = pathToDownload.renameTo(new File(localDirectoryName+filenameWithoutPrefix));
`
Regression Issue
- [ ] Select this option if this issue appears to be a regression.
Expected Behavior
Expected Behaviour: Applications should be able to download files from S3 without encountering file lock exceptions, given that each application is working with unique file paths.
Current Behavior
Problem: We are encountering FileLockException errors when 42 applications attempt to download files in parallel from S3 for processing.
Reproduction Steps
Run 42 multiples app to download files from S3, each file size ~160MB. Download them to unique folder for each app.
Possible Solution
We have a static Map that maintains file lock mappings. If a file is being downloaded and gets erased during the process, who is responsible for clearing those downloadInParallel executors? Are we calling unlock file on failure, or are we ensuring file unlocking occurs in all cases (both failure and completion)?
Additional Information/Context
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.12.589</version>
AWS Java SDK version used
1.12.589
JDK version used
java 11.0.21 2023-10-17 LTS
Operating System and version
Ubuntu
Hi @NitinPeddewad,
Thanks for reporting the issue. As you would have noticed per announcement, Java SDK v1 is in Maintenance Mode and will only get critical bug fixes and security updates. Therefore, I would suggest to validate this behavior with Java SDK v2 as well. In case you similar problem with it, kindly provide a self-contained code sample that we can use to reproduce the issue reliably.
Regards, Chaitanya
Hi @NitinPeddewad,
I am unable to reproduce the issue using below code snippet in Java SDK v1. So if you can provide a self-contained code sample that can reliably reproduce this problem, it would help for troubleshooting.
S3FileLockIssue.java
package org.example;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;
import com.amazonaws.services.s3.transfer.exception.FileLockException;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class S3FileLockIssue {
private static final String BUCKET_NAME = "<<bucket_name>>";
private static final String REGION = "us-east-1";
private static final int NUM_CONCURRENT_APPS = 42;
private static final boolean CREATE_TEST_FILES = false;
public static void main(String[] args) {
System.out.println("Starting FileLockException reproduction with " + NUM_CONCURRENT_APPS + " concurrent downloads");
if (CREATE_TEST_FILES) {
createTestFilesInS3();
}
createDirectories();
ExecutorService executor = Executors.newFixedThreadPool(NUM_CONCURRENT_APPS);
CompletableFuture<Void>[] futures = new CompletableFuture[NUM_CONCURRENT_APPS];
for (int i = 0; i < NUM_CONCURRENT_APPS; i++) {
final int appId = i;
futures[i] = CompletableFuture.runAsync(() -> {
simulateApplication(appId);
}, executor);
}
CompletableFuture.allOf(futures).join();
executor.shutdown();
try {
if (!executor.awaitTermination(60, TimeUnit.SECONDS)) {
executor.shutdownNow();
}
} catch (InterruptedException e) {
executor.shutdownNow();
}
}
private static void simulateApplication(int appId) {
System.out.println("Starting application " + appId);
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(REGION)
.withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
.build();
TransferManager transferManager = TransferManagerBuilder.standard()
.withS3Client(s3Client)
.build();
try {
for (int fileIndex = 0; fileIndex < 3; fileIndex++) {
String filename = "test-file-" + appId + "-" + fileIndex + ".parquet";
downloadFileWithRetry(transferManager, filename, appId, fileIndex);
}
} finally {
transferManager.shutdownNow();
}
}
private static void downloadFileWithRetry(TransferManager transferManager, String filename, int appId, int fileIndex) {
int maxRetries = 3;
int retryCount = 0;
while (retryCount < maxRetries) {
try {
downloadFile(transferManager, filename, appId, fileIndex);
System.out.println("App " + appId + " successfully downloaded " + filename);
break;
} catch (FileLockException e) {
retryCount++;
System.err.println("App " + appId + " FileLockException on " + filename + " (attempt " + retryCount + "): " + e.getMessage());
if (retryCount >= maxRetries) {
System.err.println("App " + appId + " failed to download " + filename + " after " + maxRetries + " attempts");
} else {
try {
Thread.sleep(1000 + (long)(Math.random() * 2000));
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
break;
}
}
} catch (Exception e) {
System.err.println("App " + appId + " unexpected error downloading " + filename + ": " + e.getMessage());
break;
}
}
}
private static void downloadFile(TransferManager transferManager, String filename, int appId, int fileIndex) throws Exception {
String bufferedDirectoryName = "app-" + appId + "-buffer/";
String localDirectoryName = "app-" + appId + "-local/";
Files.createDirectories(Paths.get(bufferedDirectoryName));
Files.createDirectories(Paths.get(localDirectoryName));
String filenameWithoutPrefix = filename;
File existingBufferedFile = new File(bufferedDirectoryName + filenameWithoutPrefix);
File existingLocalFile = new File(localDirectoryName + filenameWithoutPrefix);
if (existingBufferedFile.exists()) {
existingBufferedFile.delete();
}
if (existingLocalFile.exists()) {
existingLocalFile.delete();
}
File lockFile = new File(bufferedDirectoryName + filenameWithoutPrefix + ".lock");
if (lockFile.exists()) {
lockFile.delete();
}
File pathToDownload = new File(bufferedDirectoryName + filenameWithoutPrefix);
transferManager.download(BUCKET_NAME, filename, pathToDownload).waitForCompletion();
boolean success = pathToDownload.renameTo(new File(localDirectoryName + filenameWithoutPrefix));
if (!success) {
throw new IOException("Failed to move file from buffer to local directory");
}
}
private static void createTestFilesInS3() {
System.out.println("Creating test files in S3 bucket: " + BUCKET_NAME);
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(REGION)
.withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
.build();
try {
for (int appId = 0; appId < NUM_CONCURRENT_APPS; appId++) {
for (int fileIndex = 0; fileIndex < 3; fileIndex++) {
String filename = "test-file-" + appId + "-" + fileIndex + ".parquet";
if (!s3Client.doesObjectExist(BUCKET_NAME, filename)) {
byte[] data = new byte[160 * 1024 * 1024]; // 160MB
java.util.Arrays.fill(data, (byte) 'A');
s3Client.putObject(BUCKET_NAME, filename, new java.io.ByteArrayInputStream(data), null);
System.out.println("Created test file: " + filename);
}
}
}
System.out.println("Test files ready in S3");
} catch (Exception e) {
System.err.println("Failed to create test files: " + e.getMessage());
System.err.println("Make sure the bucket exists and you have write permissions");
}
}
private static void createDirectories() {
try {
for (int i = 0; i < NUM_CONCURRENT_APPS; i++) {
Files.createDirectories(Paths.get("app-" + i + "-buffer"));
Files.createDirectories(Paths.get("app-" + i + "-local"));
}
} catch (IOException e) {
System.err.println("Failed to create directories: " + e.getMessage());
}
}
}
Also, keep us posted if you are able to validate the issue in v2 SDK as well.
Regards, Chaitanya
One possible issue I can comprehend in FileLocks handling is that in DownloadCallable.downloadInParallel() instead doing the unlock() in finally block it is being done in a catch(Exception e) block. Which has some potential to leave a file locked causing issue in case of retry
https://github.com/aws/aws-sdk-java/blob/e1cd79b815a2b905c0a576ed5fd04c6e97a4b316/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/DownloadCallable.java#L174-L177
@NitinPeddewad Marking this to autoclose soon. As @bhoradc mentioned, we couldn't reproduce the issue, if you can send us a self-contained sample code we'll take a look.
As a reminder Java SDK v1 will reach end-of-support in December 31, 2025, we recommend to migrate to Java SDK v2.
It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}