Enable protections fails immediately after creating new security policy

[jbrmg]

Terraform Version

Terraform v1.1.9 on linux_amd64

• provider registry.terraform.io/akamai/akamai v2.0.0
• provider registry.terraform.io/hashicorp/local v2.2.3
• provider registry.terraform.io/hashicorp/null v3.1.1

Affected Resource(s)

Please list the resources as a list, for example:

• akamai_appsec_waf_protection
• akamai_appsec_api_constraints_protection

Terraform Configuration Files

resource "akamai_appsec_security_policy" "this" {
  config_id              = var.security_config_id
  security_policy_name   = random_string.security_policy.keepers.policy_name
  security_policy_prefix = random_string.security_policy.result
  default_settings       = false
}
resource "random_string" "security_policy" {
  length  = 4
  lower   = true
  upper   = false
  number  = true
  special = false

  keepers = {
    policy_name = "my_policy_name"
  }
}

resource "akamai_appsec_waf_protection" "this" {
  config_id          = var.security_config_id
  security_policy_id = akamai_appsec_security_policy.this.security_policy_id
  enabled            = true
}

resource "akamai_appsec_api_constraints_protection" "this" {
  config_id          = var.security_config_id
  security_policy_id = akamai_appsec_security_policy.this.security_policy_id
  enabled            = true
}

resource "akamai_appsec_reputation_protection" "this" {
  config_id          = var.security_config_id
  security_policy_id = akamai_appsec_security_policy.this.security_policy_id
  enabled            = true
}


resource "akamai_appsec_slowpost_protection" "this" {
  config_id          = var.security_config_id
  security_policy_id = akamai_appsec_security_policy.this.security_policy_id
  enabled            = true
}

resource "akamai_appsec_rate_protection" "this" {
  config_id          = var.security_config_id
  security_policy_id = akamai_appsec_security_policy.this.security_policy_id
  enabled            = true
}

Debug Output

2022-05-30T06:36:57.873Z [DEBUG] provider.terraform-provider-akamai_v2.0.0: 2022/05/30 06:36:57 [ERROR] calling UpdateWAFProtection: Title: Internal Server Error; Type: https://problems.luna.akamaiapis.net/appsec-configuration/error-types/INTERNAL-SERVER-ERROR; Detail: Error occurred while processing the request.: APPSEC=resourceWAFProtectionCreate OperationID=906d7195-eeaa-4d62-95e1-9a8e3f2f7214
2022-05-30T06:36:57.877Z [ERROR] vertex "module.policy_block.akamai_appsec_waf_protection.this" error: Title: Internal Server Error; Type: https://problems.luna.akamaiapis.net/appsec-configuration/error-types/INTERNAL-SERVER-ERROR; Detail: Error occurred while processing the request.

Expected Behavior

When we create a security policy, we immediately want to enable the WAF protection for the new policy. We expect that one terraform apply execution is enough to reach our goal.

Actual Behavior

The execution fails with an internal server error exception on the akamai_appsec_waf_protection resource as seen in the debug logs. If we attempt to re-run terraform apply immediately afterwards, the issue persists. After waiting for approx. 2 minutes, another terraform apply run is executed successfully.

The issue sometimes appears on other resources like akamai_appsec_api_constraints_protection as well, but most often on akamai_appsec_waf_protection.

The issue DOES NOT appear when using only the akamai_appsec_security_policy and akamai_appsec_waf_protection resources. There seems to be a race condition when modifying multiple protections at the same time.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply
2. run terraform apply again
3. wait a minute or two
4. run terraform apply again

[mgwoj]

Hello @jbrmg ,

Thank you for reporting this issue. We have started investigating it. I will inform you as soon as I get more updates.

Thanks, Michal

[kaush-]

Hi @jbrmg,

We further looked into the issue and need your help with the below points. We think what's going on is bit different. There is a known issue where trying to set multiple protections to false (as when deleting a security policy) can fail — we think there is a race condition there. This is probably what's happening here, except that you are setting multiple protections after creating a policy (instead of deleting a policy). If there is a race condition, it's between setting multiple protections — not between creating a policy and setting a protection.

Can we request you to confirm if the error happens when setting only a single protection? (And can we confirm that the issue is not that the UpdateWAFProtection request body is missing?)

Regards, Piyush

[jbrmg]

Hi @kaush- ,

thanks for the response. I can confirm that the issue does not occur when I apply only a single protection. I can also confirm that the issue is not that the UpdateWAFProtection request body is missing as I am able to enable the akamai_appsec_waf_protection successfully if it's the only protection to be enabled in a single terraform execution.

Regards Jan

[JPLodineAK]

This error occurs because Terraform by default will attempt to execute these operations in parallel, but the Open API cannot process a request to set one permission while it is in the process of setting another one. At present this situation can be avoided by either of these means:

• execute the terraform apply command with the -parallelism=1 switch to force Terraform to perform these operations in sequence rather than in parallel
• use the Terraform depends_on meta-argument on all but one of the individual protection resources, to force them to be executed in a specific sequence