Buckets Not Displayed if Not Owned by the AWS Account

[nicolasbriere1]

I am using AWS and an IAM role to connect to S3. I want to access buckets that are not owned by my AWS account. Although the IAM policy is correctly configured, the plugin has a mechanism that lists only the buckets accessible to the account:

  const bucketList = await s3Client.listBuckets({});

  const buckets =
    bucketList.Buckets?.map(b => b.Name || '')
      .filter(b => b)
      .filter(b => {
        const bucketsAllowed =
          allowedBuckets.find(a => a.platform === platform.endpointName)
            ?.buckets || [];

        // If no allowedBuckets defined for the platform, all its buckets are allowed by default
        if (bucketsAllowed.length === 0) {
          return true;
        }

        return bucketsAllowed.some(a => {
          // Add the start/end of regular expression, so no unexpected matches happen
          // Example: `test` should't match `test-one`, but `test.*` should.
          return b.match(`^${a}$`);
        });
      }) || [];

  return buckets;

The s3Client.listBuckets method only displays buckets owned by the account, and my IAM role used by Backstage is not the owner, although it has read access. I would like to create a pull request to support this use case, but I am unsure how to do so without disrupting the existing logic.

[heyLu]

I'm not sure I fully understand the use-case, but some info:

• s3Client.listBuckets is the method from the AWS SDK, so not something we control: https://github.com/spreadshirt/backstage-plugin-s3/blob/37cfe9ce4efd86188febd0f24154b02d11b75646/plugins/s3-viewer-backend/src/credentials-provider/utils.ts#L19
• that function with listBuckets is called in https://github.com/spreadshirt/backstage-plugin-s3/blob/37cfe9ce4efd86188febd0f24154b02d11b75646/plugins/s3-viewer-backend/src/credentials-provider/IAMRoleCredentialsProvider.ts#L47, so it might be possible to implement a custom CredentialsProvider for the use-case. it probably wouldn't need to be part of this plugin either, you could just add your own locally and use that.

I'm not sure if we list buckets elsewhere though, @ivangonzalezacuna should know more about that.

[nicolasbriere1]

@heyLu, yes, you're right. I can create a custom provider, but I believe this is a common use case in the AWS context (accessing buckets not owned by the current account). Perhaps we can make the IAM Role credentials provider work with this.

[ivangonzalezacuna]

Hi @nicolasbriere1. Since we don't really have a proper AWS setup to test the iam-role mode, it would be really helpful if you can help us here.

As @heyLu said, the listBuckets is called within the fetchBucketsForPlatform for each credentials provider. I believe you could try to create a custom credentials provider one based on the IAMRoleCredentialsProvider and make the adjustments to work for your use case.

If you get into that, I believe we could extend the configuration with the needed things to make it work for the use-case you describe here and also for the use-case that it currently covers.

The iam-role config can be found, and extended if needed, in this file.

Also, after our last changes we merged a few days ago, it's pretty easy to test the s3-viewer locally. You can customize the app-config.yaml as desired for your tests. And then you could make the changes to the IAMRoleCredentialsProvider directly.

Hope this info is enough for you. If not, let us know

[retrogaming457]

Hi @nicolasbriere1, I am having similar issue here as my s3 endpoint is on our local cloud. I was wondering if you finally did implement the custom provider in your case and whether it worked? if yes, can you please share what to add to which file(s)? I havent used TS before. Thanks.

[nicolasbriere1]

Hi @retrogaming457, not for the moment unfotunately, we just set a specific bucket in configuration. If you have time to work on it, i can give you a little helping hand maybe :)

[veenarm]

We built our own Provider also had to remove one of the commands that required listACL etc as our buckets don't have access to the root tenancy (not AWS, on prem provider).