Bucket name added twice in headobject command when used after call to createPresignedPost() with forcePathStyle = true

[jcpage]

Describe the bug

When I set "forcePathStyle: true" in S3ClientConfig the path in the HeadObject request adds the bucket name twice - but ONLY if I have previously used the client in a createPresignedPost(client, ...) call.

If I create two clients with identical config and use one for each op, the HeadObject call is correct. So guessing something in createPresignedPost() has an unintended side effect?

Your environment

MacOS 12.1 M1 Node.js v16.13.1 Express app (ie not a Lambda)

SDK version number

@aws-sdk/[email protected] @aws-sdk/[email protected]

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

Node v16.13.1

Steps to reproduce

Note, I am using an alternate endpoint and fake credentials for local testing. Also due to local testing, I have forcePathStyle set - if it's not set, this doesn't happen.

const config: S3ClientConfig = {
  region: `us-east-1`,
  config.endpoint = process.env.AWS_ENDPOINT
  config.credentials = {
    accessKeyId: `test`,
    secretAccessKey: `test`,
  }
  config.forcePathStyle = true
}
const s3Client = new S3(config)

...

const { url: videoUrl, fields: videoFields } =
      await createPresignedPost(s3Client, {
        Bucket: bucketName,
        Key: videoPath,
        Fields: {
          'x-amz-meta-videoid': videoId,
        },
        Conditions: [
          [`starts-with`, `$x-amz-meta-timestamp`, ``],
          [`starts-with`, `$x-amz-meta-duration`, ``],
          [`content-length-range`, 1000, 20000000],
        ],
        Expires: 600,
      })

...
In another function triggered later by S3 create object notification via SNS:

   const headArgs = {
      Key: s3Key,
      Bucket: bucketName,
    }
    const { Metadata } = await s3Client.headObject(headArgs) // *** GIVES 404

Observed behavior

The HTTP call generated from the HeadObject command has the bucket name included in the path, ie:

HEAD /BucketName/BucketName/Key

This was confirmed with Wireshark to be in the outgoing HTTP request, so not related to any details of the local testing setup.

Expected behavior

The HEAD request should just include BucketName once.

Screenshots

Not sure how useful it is, but shot of Wireshark trace...

[stophecom]

@jcpage is there a workaround for this?

[jcpage]

@jcpage is there a workaround for this?

Just what I mentioned in the report - I had to create two S3Client instances, then use one for the createPresignedPost() call and the other other calls like headObject(). It's a pretty simple workaround, I guess, but was a pretty tricky bug to track down...

[stophecom]

Thanks. I ended up doing same thing.

I was playing around with config, but didn't find anything that fixed that behavior. I'm not even sure the bug is related to forcePathStyle specifically. In my case was using getSignedUrl after createPresignedPost and ended up with an URL that had the bucket name twice, once as subdomain, once appended. e.g bucket-name.custom-endpoint.com/bucket-name, with the forcePathStyle enabled, it was custom-endpoint.com/bucket-name/bucket-name

Anyways, thanks again.

[Sceat]

It's caused by an ugly mutation inside the createPresignedPost method

    if (!client.config.bucketEndpoint) {
        endpoint.path = `/${Bucket}`;
    }

that code should be removed

[yenfryherrerafeliz]

Hi @jcpage, thanks for opening this issue and sorry for the delayed response. I can confirm this is an issue and also is caused by what @Sceat mentioned. I will mark this issue to be reviewed, so we can address this further. As workaround you could try setting the client's config's path to empty before calling headObject: Example:

const endpoint = await s3Client.config.endpoint();
endpoint.path = '';

Whole code:

import {createPresignedPost} from "@aws-sdk/s3-presigned-post";
import {S3, S3Client} from "@aws-sdk/client-s3";

const s3Client = new S3({
    region: 'REGION',
    forcePathStyle: true,
    endpoint: 'ENDPOINT'
});
let presignedPostResponse = await createPresignedPost(s3Client, {
    Bucket: 'BUCKET',
    Key: 'KEY',
    Fields: {
        'x-amz-meta-videoid': 'test-video-id',
    }
});

console.log('PreSignedPostResponse: ', presignedPostResponse);

const endpoint = await s3Client.config.endpoint();
endpoint.path = '';
const response = await s3Client.headObject({
    Bucket: 'BUCKET',
    Key: 'KEY'
});

console.log('HeadObjectResponse: ', response);

Thanks!

[Narretz]

Hi @yenfryherrerafeliz thanks for looking at this issue. Is there any progress on getting this reviewed? The fix seems pretty straightforward, and only a test is missing it seems.

[Narretz]

~~Hi @ajredniwja you are assigned to this issue, can you say something about the status here or move it forward in some way? The PR still applies it seems~~

[Narretz]

Actually, looks like this issue was fixed when the getting of the endpoint was refactored some months ago. Not sure if a test was added, though. Anyway, it works fine now for me. Afaict this issue can be closed.

[yenfryherrerafeliz]

Hi @Narretz, thanks for letting us know.

Please reach out back if you have any other issues.

Thanks!

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.