awscc_networkmanager_vpc_attachment tries to update core_network_arn attribute in-place and hangs

[joaorodrig]

Exactly the same behaviour as here: Issue 560 (https://github.com/hashicorp/terraform-provider-awscc/issues/560)

Configuration:

# Optional - Core Network
data "awscc_networkmanager_core_network" "core_network" {
  count = var.core_network_id == null ? 0 : 1
  id    = var.core_network_id
}
resource "awscc_networkmanager_vpc_attachment" "core_network" {
  count                   = var.core_network_id == null ? 0 : 1
  subnet_arns             = [ aws_subnet.private1.arn, aws_subnet.private2.arn ]
  core_network_id         = data.awscc_networkmanager_core_network.core_network[0].core_network_id
  vpc_arn                 = aws_vpc.spoke_vpc.arn
  options                 = {
    ipvipv_6_support =  false
  }
  tags                    = [
    { key = "Name", value = var.vpc_name},
    { key = "route-domain", value = var.routing_domain}
  ]
}

Behaviour:

# module.central_egress_vpc_use1[0].awscc_networkmanager_vpc_attachment.core_network[0] will be updated in-place
  ~ resource "awscc_networkmanager_vpc_attachment" "core_network" {
      + core_network_arn              = (known after apply)
        id                            = "attachment-0fb2c727206306d49"
        tags                          = [
            # (2 unchanged elements hidden)
        ]
        # (15 unchanged attributes hidden)
    }

[drewmullen]

I am able to replicate. Here is some code to help. You must execute as 2 separate root modules:

Please note there is a typo in the HCL, this is not the root cause. should be ipv_6_support not ipvipv_6_support

1st Core Network Root:

module "cloudwan" {
  source = "aws-ia/cloudwan/aws"

  global_network = {
    description = "Global Network - AWS CloudWAN Module"
  }
  core_network = {
    description            = "Core Network - AWS CloudWAN Module"
    policy_document        = data.aws_networkmanager_core_network_policy_document.main.json
  }

  tags = {
      Name = "create-global-network"
  }
}

data "aws_networkmanager_core_network_policy_document" "main" {
  core_network_configuration {
    vpn_ecmp_support = false
    asn_ranges       = ["64512-64555"]
    edge_locations {
      location = "us-east-1"
      asn      = 64512
    }
  }

  segments {
    name                          = "shared"
    description                   = "SegmentForSharedServices"
    require_attachment_acceptance = true
  }

  segment_actions {
    action     = "share"
    mode       = "attachment-route"
    segment    = "shared"
    share_with = ["*"]
  }

  attachment_policies {
    rule_number     = 1
    condition_logic = "or"

    conditions {
      type     = "tag-value"
      operator = "equals"
      key      = "segment"
      value    = "shared"
    }
    action {
      association_method = "constant"
      segment            = "shared"
    }
  }
}

2nd VPC Attachment Root

module "vpc" {
  source   = "aws-ia/vpc/aws"
  version = ">= 1.0.0"

  name       = "multi-az-vpc"
  cidr_block = "10.0.0.0/20"
  az_count   = 2

  subnets = {
    private = {
      netmask      = 24
    }
  }
}

resource "awscc_networkmanager_vpc_attachment" "core_network" {
  subnet_arns             = [ for _, value in module.vpc.private_subnet_attributes_by_az: value.arn]
  core_network_id         = var.core_network_id
  vpc_arn                 = module.vpc.vpc_attributes.arn
  options                 = {
    ipv_6_support =  false
  }
  tags                    = [
    { key = "Name", value = "multi-az-vpc"},
    { key = "segment", value = "shared"}
  ]
}

variable "core_network_id" {}

[drewmullen]

I assume this has something to do with the schema regarding ARN vs ID:

error:

awscc_networkmanager_vpc_attachment.core_network will be updated in-place
  ~ resource "awscc_networkmanager_vpc_attachment" "core_network" {
      + core_network_arn              = (known after apply)
        id                            = "attachment-<>"
        tags                          = [
            # (2 unchanged elements hidden)
        ]
        # (15 unchanged attributes hidden)
    }

[drewmullen]

The offending ARN is readonly:

    "readOnlyProperties": [
        "/properties/CoreNetworkArn",

@ewbankkit does it seem like this is service side or provider side?

[drewmullen]

i also noticed that the resource required acceptance. once i accepted the problem persisted

terraform state show awscc_networkmanager_vpc_attachment.core_network
# awscc_networkmanager_vpc_attachment.core_network:
resource "awscc_networkmanager_vpc_attachment" "core_network" {
   ...
    state                         = "PENDING_ATTACHMENT_ACCEPTANCE"

[ewbankkit]

(known after apply) for core_network_arn is caused I think by the JSON policy diff issue: https://github.com/hashicorp/terraform-provider-awscc/issues/509. If you replace

  core_network = {
    description            = "Core Network - AWS CloudWAN Module"
    policy_document        = data.aws_networkmanager_core_network_policy_document.main.json
  }

with

  core_network = {
    description            = "Core Network - AWS CloudWAN Module"
    policy_document        = jsonencode(jsondecode(data.aws_networkmanager_core_network_policy_document.main.json))
  }

this update should disappear.

[jeanguanda]

Hello everyone, I face the same issue about an attachment trying to update core_network_arn all time and if I try to apply, just hangs.

I'm already using: policy_document = jsonencode(jsondecode(data.aws_networkmanager_core_network_policy_document.core_policy.json))

[avedismanifold]

while this update in place is its own issue, the behavior is causing a hang seems to be a bug with aws cloudcontrol. when you pass in a patch document of "[]" it immediately returns success, then when you query the request, it is stuck in pending. You can replicate this with cli:

aws cloudcontrol update-resource --type-name AWS::NetworkManager::CoreNetwork --identifier core-network-xxxxx --patch-document "[]"

{
    "ProgressEvent": {
        "TypeName": "AWS::NetworkManager::CoreNetwork",
        "Identifier": "core-network-xxxxx",
        "RequestToken": "23xxxxx07",
        "Operation": "UPDATE",
        "OperationStatus": "SUCCESS",
        ...
        

aws cloudcontrol get-resource-request-status --request-token 23xxxxx07

{
    "ProgressEvent": {
        "TypeName": "AWS::NetworkManager::CoreNetwork",
        "RequestToken": "23xxxxx07",
        "OperationStatus": "PENDING",
        "EventTime": "2022-09-07T16:39:46.821000+00:00"
    }
}

This hangs indefinitely.

On terraform, you can check out debug log to verify that this is indeed what is happening.

Made a post about this: https://repost.aws/questions/QUI53s7_TxSz-oZkZmMZ4q8Q/bug-cloudcontrol-on-networkmanager-core-network-with-empty-policy-will-hang

NOTE: I did not verify that this is the root cause for this ticket, but I did verify that this is the root cause for the similar issue: https://github.com/hashicorp/terraform-provider-awscc/issues/560

EDIT: I have verified that this is the root cause of this ticket as well

[avedismanifold]

You can get around this issue by making any change to the resource that will enable terraform to send a nonempty patch document to cloud control.

For instance, if you add a tag to the resource, then it will no longer pass "[]"

obviously not a great solution.

Personally what I am going to do is to simply split the wan/core network into its own root and likewise with the network attachments. Then I am going to simply hardcode the core network id in. Im pretty sure this will work, but I will update once I test

EDIT: the above did not work, the attachment, despite having a fixed core network ID, was trying to update the arn.

Instead, I will add a tag with the value timestamp() so that it changes on every apply :(

[drewmullen]

i retried this exact stack with the jsonencode(jsondecode()) on the core_network resource and i no longer have issues with faux diffs on vpc_attachment

$ tf -v
Terraform v1.2.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v4.33.0
+ provider registry.terraform.io/hashicorp/awscc v0.33.0

[ewbankkit]

That is likely because of the merge of https://github.com/hashicorp/terraform-provider-awscc/pull/667.