google_storage_transfer_job permanent diff

[xM8WVqaG]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

tf --version
Terraform v1.1.8
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v4.18.0

Affected Resource(s)

• google_storage_transfer_job

Terraform Configuration Files

resource "google_storage_transfer_job" "this" {
  description = "bucket-name description"
  project     = "gcp_project_id"
  status      = "ENABLED"

  transfer_spec {
    transfer_options {
      delete_objects_from_source_after_transfer  = false
      delete_objects_unique_in_sink              = true
      overwrite_objects_already_existing_in_sink = true
    }

    aws_s3_data_source {
      bucket_name = "bucket-name"

      aws_access_key {
        access_key_id     = "aws_iam_access_key_id"
        secret_access_key = "aws_iam_access_key_secret"
      }
    }
  }
}

Debug Output

Drift Output

# google_storage_transfer_job.this has changed
~ resource "google_storage_transfer_job" "this" {
    - deletion_time          = "" -> null
      id                     = "bucket-name/123"
    ~ last_modification_time = "2022-04-19T09:16:44.198468187Z" -> "2022-04-19T09:19:10.026136074Z"
      name                   = "transferJobs/123"
      # (4 unchanged attributes hidden)


    ~ transfer_spec {
        ~ aws_s3_data_source {
            - role_arn    = "" -> null
              # (1 unchanged attribute hidden)

            - aws_access_key {
                - access_key_id     = (sensitive value)
                - secret_access_key = (sensitive value)
              }
          }


          # (2 unchanged blocks hidden)
      }
      # (1 unchanged block hidden)
  }

Plan Output

# google_storage_transfer_job.this will be updated in-place
~ resource "google_storage_transfer_job" "this" {
    + deletion_time          = ""
      id                     = "bucket-name/123"
      name                   = "transferJobs/123"
      # (5 unchanged attributes hidden)


    ~ transfer_spec {
        ~ aws_s3_data_source {
            + role_arn    = ""
              # (1 unchanged attribute hidden)

            + aws_access_key {
                + access_key_id     = (sensitive value)
                + secret_access_key = (sensitive value)
              }
          }


          # (2 unchanged blocks hidden)
      }
      # (1 unchanged block hidden)
  }

Expected Behavior

When google_storage_transfer_job.transfer_spec.aws_s3_data_source.role_arn is not set, the Google Cloud provider shouldn't get caught up always suggesting it changes the value. It also shouldn't care about setting google_storage_transfer_job.deletion_time as I understand that's a read-only attribute.

Actual Behavior

During every plan the Google Cloud provider detects drift for google_storage_transfer_job.deletion_time and google_storage_transfer_job.transfer_spec.aws_s3_data_source.role_arn from "" to null and then attempts to set them to "".

Steps to Reproduce

1. terraform apply

Important Factoids

References

• https://github.com/hashicorp/terraform-provider-google/pull/10950
• https://github.com/hashicorp/terraform-provider-google/issues/10967

b/302797561

[psvaiter]

I'm stuck in version 4.8.0 because since version 4.9.0 this bug is present and I don't want to pollute the tf plan output with this permadiff at every change. OP has referenced the issue that introduced the bug.

[psvaiter]

The provider version is already in version 4.28.0 and the bug introduced in version 4.9.0 is still there. Previous to 4.9.0 it was working as expected. Shouldn't this be prioritized? @c2thorn

[psvaiter]

@yudoufu I saw you made a PR recently to fix a permadiff related to this resource. Could you help me with this one?

[c2thorn]

I'll take a look at this

[c2thorn]

not sure what is going on with the deletion_time field, as that is an output only field, and was unable to replicate. Have a PR that should resolve the role_arn diff. Due to timing, it will likely be a couple of releases (maybe 4.36?) out.

In the meantime, you should be able to stop the diffs with lifecycle ignore_changes argument on the resource like:

  lifecycle {
    ignore_changes = [
      transfer_spec.0.aws_s3_data_source.0.aws_access_key,
      transfer_spec.0.aws_s3_data_source.0.role_arn,
    ]
  }

And only removing it if you need to update the fields.

[psvaiter]

@c2thorn I think that changing the role_arn to computed will not solve the permadiff on aws_access_key which is what I'm using. After this change https://github.com/hashicorp/terraform-provider-google/pull/10950 the keys are not being persisted in state.

Here's what's happening:

• First it detects changes "made outside"
• Then it tries to apply the key again

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # module.ingestion.module.data_transfer["composition"].google_storage_transfer_job.load_from_s3[0] has changed
  ~ resource "google_storage_transfer_job" "load_from_s3" ***
      - deletion_time          = "" -> null
        id                     = "(redacted)"
      ~ last_modification_time = "2022-08-31T19:21:06.844403992Z" -> "2022-08-31T19:55:31.137141444Z"
        name                   = "transferJobs/8173655488523024047"
        # (4 unchanged attributes hidden)


      ~ transfer_spec ***
          ~ aws_s3_data_source ***
              - role_arn    = "" -> null
                # (1 unchanged attribute hidden)

              - aws_access_key ***
                  - access_key_id     = (sensitive value)
                  - secret_access_key = (sensitive value)
                ***
            ***


            # (3 unchanged blocks hidden)
        ***
        # (1 unchanged block hidden)
    ***


[...]


Terraform will perform the following actions:
# module.ingestion.module.data_transfer["composition"].google_storage_transfer_job.load_from_s3[0] will be updated in-place
  ~ resource "google_storage_transfer_job" "load_from_s3" ***
      + deletion_time          = ""
        id                     = "(redacted)"
        name                   = "transferJobs/8173655488523024047"
        # (5 unchanged attributes hidden)


      ~ transfer_spec ***
          ~ aws_s3_data_source ***
              + role_arn    = ""
                # (1 unchanged attribute hidden)

              + aws_access_key ***
                  + access_key_id     = (sensitive value)
                  + secret_access_key = (sensitive value)
                ***
            ***



            # (3 unchanged blocks hidden)
        ***
        # (1 unchanged block hidden)
    ***

[c2thorn]

@psvaiter I see. I believe the API does not return values for aws_access_key (likely because they are sensitive), and the provider should be updated to ignore that fact.

I am not able to reproduce the role_arn or deletion_time diffs, can you verify if adding

  lifecycle {
    ignore_changes = [
      transfer_spec.0.aws_s3_data_source.0.aws_access_key,
    ]
  }

to the resource prevents all 3 diffs? If not, then how about the version in https://github.com/hashicorp/terraform-provider-google/issues/11535#issuecomment-1234603137 that also adds transfer_spec.0.aws_s3_data_source.0.role_arn ?

[psvaiter]

@c2thorn Just adding the transfer_spec.0.aws_s3_data_source.0.aws_access_key prevent all 3 diffs.

I'll keep this block for now in order make the output clean, however we may need to update credentials in the future and a fix will be good.

I believe it's not an API problem as the version 4.8.0 works as expected.

[mcmarkj]

Hi - are there any updates around a fix for this?

[m0ps]

permanent diff exists is still exists:

  # google_storage_transfer_job.cx_s3_backup will be updated in-place
  ~ resource "google_storage_transfer_job" "cx_s3_backup" {
      + deletion_time          = ""
        id                     = "project_id/1111111111111111111"
        name                   = "transferJobs/222222222222222"
        # (5 unchanged attributes hidden)

      ~ transfer_spec {
          + sink_agent_pool_name   = ""
          + source_agent_pool_name = ""

          ~ aws_s3_data_source {
              + path        = ""
              + role_arn    = ""
                # (1 unchanged attribute hidden)

              + aws_access_key {
                  + access_key_id     = (sensitive value)
                  + secret_access_key = (sensitive value)
                }
            }

            # (2 unchanged blocks hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

[googlyrahman]

Hi, I'm able to reproduce this issue on the mentioned version, but not on the latest version of the provider. Can you upgrade it

[rworley-monster]

I still see this permadiff after applying 5.28.0. In our case, it only wants to change:

      ~ transfer_spec {
          ~ aws_s3_data_source {
                # (1 unchanged attribute hidden)

              + aws_access_key {
                  + access_key_id     = (sensitive value)
                  + secret_access_key = (sensitive value)
                }
            }

            # (1 unchanged block hidden)
        }

[googlyrahman]

Hi, That's correct - i'm also able to see this on latest version, I've raised a PR for fixing the above.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.