Additional S3 Bucket configurations seems to not update Bucket state properly to reflect AWS

[jesperseverinsen]

What happened?

Using the Automation API, creating stacks in an AWS S3 Bucket, it seems like resources such as aws.s3.BucketServerSideEncryptionConfigurationV2, aws.s3.BucketVersioningV2, aws.s3.BucketPolicy and aws.s3.BucketLoggingV2 is not properly updating the state of the Bucket in Pulumi, to reflect thestate of the bucket in AWS.

An example is the versioning, where after deploying using the Pulumi Automation API, the output of the S3 Bucket contains

"outputs": {
    ... ,
    "versionings": [
        {
            "enabled": false,
            "mfaDelete": false
        }
    ],
    ... ,
},

with the BucketVersioningV2 resource having the following input and output:

"inputs": {
    "__defaults": [],
    "bucket": <bucket>,
    "versioningConfiguration": {
        "__defaults": [],
        "status": "Enabled"
    }
},
"outputs": {
    "bucket": <bucket>,
    "expectedBucketOwner": "",
    "id": <id>,
    "versioningConfiguration": {
        "mfaDelete": "",
        "status": "Enabled"
    }
},

However, running refresh on the stack results in updating the bucket resource itself (updated (2s) [diff: ~loggings,serverSideEncryptionConfigurations,versionings]), with the output in Pulumi being updated to properly reflect the state of the bucket in AWS:

"versionings": [
    {
        "enabled": true,
        "mfaDelete": false
    }
],

Example

const bucket = new aws.s3.BucketV2(
  `${name}-s3-bucket`,
  {},
  { parent: this }
);

new aws.s3.BucketServerSideEncryptionConfigurationV2(
  `${name}-s3-bucket-encryption-configuration`,
  {
    bucket: bucket.id,
    rules: [
      {
        applyServerSideEncryptionByDefault: {
          kmsMasterKeyId: args.kmsAliasArn,
          sseAlgorithm: 'aws:kms',
        },
      },
    ],
  },
  { parent: this }
);

new aws.s3.BucketVersioningV2(
  `${name}-s3-bucket-versioning`,
  {
    bucket: bucket.id,
    versioningConfiguration: { status: 'Enabled' },
  },
  { parent: this }
);

new aws.s3.BucketPolicy(
  `${name}-s3-bucket-logging-policy`,
  {
    bucket: args.logBucket,
    policy: loggingPolicyDocument.json,
  },
  { parent: this }
);

new aws.s3.BucketLoggingV2(
  `${name}-s3-bucket-logging`,
  { ...loggingConfig },
  { parent: this }
);

Output of pulumi about

CLI Version 3.94.2 Go Version go1.21.4 Go Compiler gc

Host OS darwin Version 13.6 Arch arm64

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

I am sorry this is not working as expected. This is a very interesting issue, thank you @jesperseverinsen for filing this with a repro, my team will have a look as time permits.

[t0yv0]

Still reproduces on latest unfortunately.

import * as aws from "@pulumi/aws";

const name = "aws-3166";

const bucket = new aws.s3.BucketV2(`${name}-s3-bucket`);

new aws.s3.BucketVersioningV2(
  `${name}-s3-bucket-versioning`,
  {
    bucket: bucket.id,
    versioningConfiguration: { status: 'Enabled' },
  },
);

// refresh yield this diff:
//
// pulumi:pulumi:Stack: (same)
//   [urn=urn:pulumi:dev::aws-3166::pulumi:pulumi:Stack::aws-3166-dev]
//   ~ aws:s3/bucketV2:BucketV2: (update)
//       [id=aws-3166-s3-bucket-3925e83]
//       [urn=urn:pulumi:dev::aws-3166::aws:s3/bucketV2:BucketV2::aws-3166-s3-bucket]
//       [provider=urn:pulumi:dev::aws-3166::pulumi:providers:aws::default_6_35_0::504bf98f-fbe4-47c0-a038-358bcb6735d8]
//       --outputs:--
//     ~ versionings                       : [
//         ~ [0]: {
//                 ~ enabled  : false => true
//                   mfaDelete: false
//               }
//       ]

[t0yv0]

import * as aws from "@pulumi/aws";

const example = new aws.s3.BucketV2("example", {bucket: "my-tf-example-bucket-anton-2024-05-14"});

const exampleBucketOwnershipControls = new aws.s3.BucketOwnershipControls("example", {
    bucket: example.id,
    rule: {
        objectOwnership: "BucketOwnerPreferred",
    },
});

const exampleBucketPublicAccessBlock = new aws.s3.BucketPublicAccessBlock("example", {
    bucket: example.id,
    blockPublicAcls: false,
    blockPublicPolicy: false,
    ignorePublicAcls: false,
    restrictPublicBuckets: false,
});

const exampleBucketAclV2 = new aws.s3.BucketAclV2("example", {
    bucket: example.id,
    acl: "public-read",
}, {
    dependsOn: [
        exampleBucketOwnershipControls,
        exampleBucketPublicAccessBlock,
    ],
});

A bit of the same problem here with ACL and ownership controls competing with the grants property.

    ~ aws:s3/bucketV2:BucketV2: (update)
        [id=my-tf-example-bucket-anton-2024-05-14]
        [urn=urn:pulumi:dev::2024-05-14::aws:s3/bucketV2:BucketV2::example]
        [provider=urn:pulumi:dev::2024-05-14::pulumi:providers:aws::default_6_35_0::e52ec717-9340-40f3-8505-5e9ddd5d1c17]
        --outputs:--
      ~ grants                            : [
            [0]: {
                    id         : "e07865a5679c7977370948f1f1e51c21b12d8cfdd396a7e3172275d9164e01b8"
                    permissions: [
                        [0]: "FULL_CONTROL"
                    ]
                    type       : "CanonicalUser"
                    uri        : ""
                }
          + [1]: {
                  + id         : ""
                  + permissions: [
                  +     [0]: "READ"
                    ]
                  + type       : "Group"
                  + uri        : "http://acs.amazonaws.com/groups/global/AllUsers"
                }
        ]

[vedantrathore]

Hi, wanted to check if this is being worked on. We are facing into this problem as well where pulumi refresh sees that bucket configuration is not updated and deletes that configuration it in subsequent runs.

[t0yv0]

This behavior is also present in Terraform with -refresh=false but is masked by Terraform performing refresh by default on update, whereas Pulumi intentionally tries to mimic the -refresh=false behavior. You may be able to work around by pulumi up --refresh.

We are not working on this bug on the Pulumi end unfortunately and I will close this as won't fix. Our understanding is that aws.s3.BucketV2 state properties such as ones pertaining to versioningConfiguration are deprecated and upstream is intending to remove them in the next major version of the provider. At that point pulumi-aws will also release a matching version update. After BucketV2 properties are removed, refresh will only be updating the state of side-by-side resources such as BucketVersioningV2 which should resolve the issue here. Until that time we recommend working around by running refresh manually.