[S3] Bucket name containing slash character fails with any operation

[jereztech]

Describe the bug

After migrating SDKv1 to SDKv2, accessing any bucket resource with any operation: create, upload, list, delete, etc. containing slash character "/" in bucket name fails throwing the following exception: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Example: Uploading objects to the root of the bucket works, but when we try to save it to a subfolder it fails: Uploading object to "my-bucket" OK Uploading object to "my-bucket/subfolder" FAILS - Note that slash is present!

Expected Behavior

Bucket names with subfolders should work.

Current Behavior

software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Reproduction Steps

String bucketName = "my-bucket/subfolder", objectName = "123.jpg";
s3TransferManager.upload(builder -> builder.requestBody(AsyncRequestBody.fromBytes(objectBytes))
.putObjectRequest(request -> request.bucket(bucketName).key(objectName)));

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.17.228

JDK version used

1.8.0_321

Operating System and version

macOS Monterey v12.4

[yasminetalby]

Hello @jereztech, Thank you very much for documenting this issue, I was able to reproduce the error. There is two case scenarios based on what you are trying to accomplish.

1. If you are looking to name your bucket with a / character in the name, this is not allowed within the S3 bucket naming convention. To solve this you would have to use proper naming convention as defined here : https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html

2. If you are trying to access a subfolder within a bucket the proper syntax to do so would be to specify the subfolder within the key value. To solve this in the code you have provided you would have to change the line String bucketName = "my-bucket/subfolder", objectName = "123.jpg"; to String bucketName = "my-bucket", objectName = "subfolder/123.jpg";

Let me know if this solves the issue encountered. Best, Yasmine.

[jereztech]

Hi @yasminetalby ok, this solves issues with object operations, but makes it difficult to work with subfolders in bucket-based operations, for example, if we organize the resources with REST style i.e. "my-bucket/users/id-123/docs" it is impossible to exclude only the docs of a certain user due to SDKv2 regression. Please note that this works with SDKv1!

Another problem is the error message, it is generic and does not offer relevant information to identify issues. All the errors that I have obtained in SDKv2 have the same message despite being different problems. This must improve!

[yasminetalby]

Hello @jereztech, Thank you for your answer and feedback!

1. Could you provide an example of limitation on the bucket subfolder based operations? Most operations supported for bucket are supported for object. Since subfolders as considered as objects you can use the object based operation and specify the key of the subfolder (or subsub[...]folder) you would like the operation to be conducted on.

Example: if you would like to delete the file.txt object within my-bucket/users/id-123/docs/file.txt, you would need to do the following:

String BucketName = "my-bucket";
        String FileName = "users/id-123/docs/file.txt";

        DeleteObjectRequest request = DeleteObjectRequest.builder()
                .bucket(BucketName)
                .key(FileName)
                .build();
        client.deleteObject(request);

_Note: when deleting an object within a subfolder, if the subfolder is empty after deletion of the object it will be automatically removed. _

2. The error message you have encountered reflects the service side error caused by a misuse of the Java SDK through the addition of the / character correctly. We understand how this might be limiting in your investigation. The Java DSK v2 covers best practice and guideline ressources that can help: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html . Furthermore, if you have any additional questions regarding the usage of an AWS Java SDK service, we do have a discussion section where we provide support : https://github.com/aws/aws-sdk-java-v2/discussions

I hope this helps, let me know if you have any additional questions. Best,

Yasmine

[jereztech]

Hello @yasminetalby my goal is not to delete files one by one but the entire subfolder, let's see some use cases where this is essential:

Use Case 1 - Exclude User Account: The system should exclude all the files of the user who requests the exclusion of his/her account. The resources are organized as follows:

• corp-bucket/users/id-123/avatars/file1-1K.jpg
• corp-bucket/users/id-123/docs/file1-1K.txt
• corp-bucket/users/id-123/contacts/id-456/file1-1K.txt
• corp-bucket/users/id-123/chats/id-456/file1-1K.jpg
• corp-bucket/users/id-123/jobs/id-789/file1-1K.txt
• and many others...

Imagine that the user is active 15 years ago with approximately 1 million files saved in S3, it would be impractical, tedious and inefficient to have to list each file and exclude them one by one. In this case the subfolder corp-bucket/users/id-123 must be deleted without any extra effort.

Use Case 2 - Customer migration from datacenter us-east-1 to us-east-2: Let's imagine that datacenter us-east-1 will go out of operation and all customers must be migrated to datacenter us-east-2. To fulfill the goal the subfolder amazon/datacenters/us-east-1 with 100 trillion files must be copied to amazon/datacenters/us-east-2 and then excluded. In this case we have an unpredictable complex tree of dependencies where each node is a subfolder, the exclusion to be efficient must be at the root node, consequently, the subfolder amazon/datacenters/us-east-1 must be deleted without any extra effort. Again, it would be impractical, tedious and inefficient to have to list each file and exclude them one by one.

  import org.springframework.stereotype.Service;
  import software.amazon.awssdk.services.s3.S3Client;
  import software.amazon.awssdk.services.s3.model.DeleteBucketRequest;
  
  @Service
  public class AmazonS3Client {
  
      private final S3Client s3 = S3Client.create();
  
      public void deleteBucket(String bucketName) {// bucketName = "corp-bucket/users/id-123";
	      s3.deleteBucket(DeleteBucketRequest.builder().bucket(bucketName).build());
	      s3.close();
      }
  
  }

This is a regression, it was working perfectly with SDKv1!

[yasminetalby]

Hello @jereztech ,

Thank you very much for the clarification and the detailed explanation of use cases. This is a feature request for the Java SDKv2. I will bring this up to the team and will keep you updated.

Best,

Yasmine.

[yasminetalby]

Hello @jereztech ,

Update: This feature request is in our backlog. I will be updating your submission to a feature request

Thank you for your submission. Best,

Yasmine

[maciejwalkowiak]

@yasminetalby does this board reflect actual team backlog?

[yasminetalby]

Hello @maciejwalkowiak ,

Unfortunately, this board does not reflect the current team backlog. We are in the process of updating the team backlog display on GitHub. If this is a feature you would like to see implemented you can showcase your interest by adding a thumbs up emoji 👍🏼 on the feature request.

Best regards,

Yasmine