terraform-provider-ibm
terraform-provider-ibm copied to clipboard
Published
20 hours ago •
IBM-Cloud
IBM-Cloud
Race condition in creation using ibm_container_worker_pool
Terraform Version Terraform: 1.0.11
- ibm-cloud/ibm v1.37.1
Affected resource(s)
- ibm_container_worker_pool
Terraform Configuration Files
resource "ibm_container_worker_pool" "worker_pool_autoscale" {
for_each = local.worker_pools_autoscale
depends_on = [ibm_container_cluster.cluster, ibm_container_cluster.cluster_autoscale]
// WARNING Any changes here also change in worker_pool
cluster = local.cluster_name
resource_group_id = local.resource_group_id
worker_pool_name = each.key
machine_type = each.value.machine_type
size_per_zone = each.value.size_per_zone
hardware = each.value.hardware
labels = merge(local.metadata.labels, each.value.labels)
dynamic "timeouts" {
for_each = local.timeouts
content {
update = timeouts.value.update
}
}
lifecycle {
ignore_changes = [size_per_zone]
}
}
Expected behaviour
Worker pools created and ibm_container_worker_pool_zone_attachment creates worker nodes under newly created worker pools.
Actual behaviour
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["monitoring"]: Creation complete after 41s [id=b2c-auth-stage-iks-dal-01/c6pnv6od0ur22qasfv10-68cf5d1]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Still creating... [50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra"]: Still creating... [50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Still creating... [1m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra"]: Still creating... [1m0s elapsed]
Error: Request failed with status code: 409, ServerErrorResponse: {"incidentID":"ebc4f26a-6d76-4f49-8724-306d3b8f2586,ebc4f26a-6d76-4f49-8724-306d3b8f2586","code":"E0d39","description":"A worker pool with the same name already exists within the cluster. Choose another name.","type":"Provisioning"}
with module.cluster.ibm_container_worker_pool.worker_pool["cassandra"],
on .terraform/modules/cluster/modules/ibm/iks-cluster/main.tf line 78, in resource "ibm_container_worker_pool" "worker_pool":
78: resource "ibm_container_worker_pool" "worker_pool" {
Error: Request failed with status code: 409, ServerErrorResponse: {"incidentID":"1502248a-2708-4a7c-b7df-2a5080e25df8,1502248a-2708-4a7c-b7df-2a5080e25df8","code":"E0d39","description":"A worker pool with the same name already exists within the cluster. Choose another name.","type":"Provisioning"}
with module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"],
on .terraform/modules/cluster/modules/ibm/iks-cluster/main.tf line 99, in resource "ibm_container_worker_pool" "worker_pool_autoscale":
99: resource "ibm_container_worker_pool" "worker_pool_autoscale" {
Steps to reproduce
- create cluster with
ibm_container_worker_poolhaving multiple worker pools defined, amount of worker pools> 3
Additional info
ibm_container_worker_pool works properly if it creates only one worker pool.
Can you please share us the terraform plan and the local.worker_pools_autoscale output
@hkantare put plan output and variable output into gist: https://gist.github.com/DenysChekirda/38a0c11ff31499fd575e5a8bb624f7b5
Hi @hkantare - is there anything else you need from @DenysChekirda on this? Thanks for all you do!
In the above output we can't see any info releted to local.worker_pools_autoscale
resource "ibm_container_worker_pool" "worker_pool_autoscale" {
for_each = local.worker_pools_autoscale
depends_on = [ibm_container_cluster.cluster, ibm_container_cluster.cluster_autoscale]
// WARNING Any changes here also change in worker_pool
cluster = local.cluster_name
resource_group_id = local.resource_group_id
worker_pool_name = each.key
machine_type = each.value.machine_type
size_per_zone = each.value.size_per_zone
hardware = each.value.hardware
labels = merge(local.metadata.labels, each.value.labels)
dynamic "timeouts" {
for_each = local.timeouts
content {
update = timeouts.value.update
}
}
lifecycle {
ignore_changes = [size_per_zone]
}
}
Can you share the details of local.worker_pools_autoscale
Hi @hkantare as for the local.worker_pools_autoscale - output can be found starting of this line in gist: https://gist.github.com/DenysChekirda/38a0c11ff31499fd575e5a8bb624f7b5#file-plan-tf-L609
This is how we are generating this variable
worker_pools_autoscale = {
for pool, value in var.extra_worker_pools :
pool => value
if value.autoscaled == true
}
Where var.extra_worker_pools
variable "extra_worker_pools" {
type = map(
object({
machine_type = string
hardware = string
size_per_zone = number
vlans = list(string)
labels = map(string)
autoscaled = bool
upgrade_workers = bool
})
)
default = {}
}
This problem were seen with in specific region for some period of time. I could not say that this is persistent problem so I would say it is intermittent.
@hkantare okay, we just hit another variation of this on 2/23. In this case we changed one worker pool's machine type (NOTE: as this is a non-production cluster we did not care that the change would cause an outage).
The plan looked like this:
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"] must be replaced
-/+ resource "ibm_container_worker_pool" "worker_pool_autoscale" {
~ id = "b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1" -> (known after apply)
~ machine_type = "b3c.4x16" -> "c3c.16x16" # forces replacement
+ region = (known after apply)
~ resource_controller_url = "https://cloud.ibm.com/kubernetes/clusters" -> (known after apply)
~ size_per_zone = 2 -> 1
~ state = "active" -> (known after apply)
~ worker_pool_id = "c5e00fkd055e7mq0rkr0-684fbd1" -> (known after apply)
~ zones = [
- {
- private_vlan = "3149246"
- public_vlan = "3149248"
- worker_count = 7
- zone = "dal10"
},
- {
- private_vlan = "3149244"
- public_vlan = "3149250"
- worker_count = 7
- zone = "dal12"
},
] -> (known after apply)
# (6 unchanged attributes hidden)
}
Plan: 1 to add, 0 to change, 1 to destroy.
Our apply looked like this (NOTICE: The zones are NOT attached):
terraform apply -no-color -auto-approve -input="false" terraform.tfplan
----------------
[Pipeline] sh
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 1m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 1m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 1m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 1m30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 1m40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 1m50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 2m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 2m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 2m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 2m30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 2m40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 2m50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 3m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 3m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 3m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 3m30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 3m40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 3m50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 4m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 4m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 4m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 4m30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 4m40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 4m50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 5m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 5m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 5m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 5m30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 5m40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 5m50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 6m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 6m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 6m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 6m30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 6m40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 6m50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 7m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 7m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 7m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 7m30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 7m40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 7m50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 8m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 8m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still destroying... [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-684fbd1, 8m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Destruction complete after 8m27s
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Creating...
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still creating... [10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Creation complete after 16s [id=b2c-auth-qa-iks-dal-01/c5e00fkd055e7mq0rkr0-e2e319f]
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
To highlight this issue, we did a plan IMMEDIATELY after the above apply and saw this:
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# module.cluster.ibm_container_worker_pool_zone_attachment.worker_pool_zone_attachment["alb/dal10"] will be created
+ resource "ibm_container_worker_pool_zone_attachment" "worker_pool_zone_attachment" {
+ cluster = "b2c-auth-qa-iks-dal-01"
+ id = (known after apply)
+ private_vlan_id = "3149246"
+ public_vlan_id = "3149248"
+ region = (known after apply)
+ resource_group_id = "657d29fdf24f4bec816e10a43c47c032"
+ wait_till_albs = true
+ worker_count = (known after apply)
+ worker_pool = "alb"
+ zone = "dal10"
}
# module.cluster.ibm_container_worker_pool_zone_attachment.worker_pool_zone_attachment["alb/dal12"] will be created
+ resource "ibm_container_worker_pool_zone_attachment" "worker_pool_zone_attachment" {
+ cluster = "b2c-auth-qa-iks-dal-01"
+ id = (known after apply)
+ private_vlan_id = "3149244"
+ public_vlan_id = "3149250"
+ region = (known after apply)
+ resource_group_id = "657d29fdf24f4bec816e10a43c47c032"
+ wait_till_albs = true
+ worker_count = (known after apply)
+ worker_pool = "alb"
+ zone = "dal12"
}
Plan: 2 to add, 0 to change, 0 to destroy.
The next apply we did created the zone attachments.
Hi @hkantare we faced this issue once again on different account.
From our terraform modules code perspective - nothing been changed. We are using stable versions for last 7-9 months.
IBM Terraform provider version: v1.39.0 Terraform version: 1.1.6
Plan output https://gist.github.com/DenysChekirda/23d81455cf3c77c21c35138132a31a7d
Output log
module.pg-umbx.random_password.adminpassword[0]: Creating...
module.pg-umsn.random_password.adminpassword[0]: Creating...
module.pg-umwu.random_password.userspassword["sun-ms-auth-user-mgt"]: Creating...
module.pg-umwu.random_password.adminpassword[0]: Creating...
module.pg-umsn.random_password.userspassword["sun-ms-auth-user-mgt"]: Creating...
module.pg-umsn.random_password.adminpassword[0]: Creation complete after 0s [id=none]
module.pg-umsn.random_password.userspassword["sun-ms-auth-user-mgt"]: Creation complete after 0s [id=none]
module.pg-umsn.random_password.userspassword["postgres-exporter"]: Creating...
module.pg-umbx.random_password.adminpassword[0]: Creation complete after 0s [id=none]
module.pg-umwu.random_password.userspassword["sun-ms-auth-user-mgt"]: Creation complete after 0s [id=none]
module.pg-umwu.random_password.adminpassword[0]: Creation complete after 0s [id=none]
module.pg-umsn.random_password.userspassword["postgres-exporter"]: Creation complete after 0s [id=none]
module.pg-umwu.random_password.userspassword["postgres-exporter"]: Creating...
module.pg-umbx.random_password.userspassword["postgres-exporter"]: Creating...
module.pg-umbx.random_password.userspassword["sun-ms-auth-user-mgt"]: Creating...
module.pg-umwu.random_password.userspassword["postgres-exporter"]: Creation complete after 0s [id=none]
module.pg-umbx.random_password.userspassword["postgres-exporter"]: Creation complete after 0s [id=none]
module.pg-umbx.random_password.userspassword["sun-ms-auth-user-mgt"]: Creation complete after 0s [id=none]
module.vlan-map.ibm_network_vlan.managed_private["dal10"]: Creating...
module.cos-resource.ibm_resource_instance.cos-resource-instance: Creating...
module.vlan-map.ibm_network_vlan.managed_private["dal12"]: Creating...
module.pg-umsn.ibm_database.database: Creating...
module.pg-umwu.ibm_database.database: Creating...
module.pg-umbx.ibm_database.database: Creating...
module.vlan-map.ibm_network_vlan.managed_private["dal10"]: Still creating... [10s elapsed]
module.cos-resource.ibm_resource_instance.cos-resource-instance: Still creating... [10s elapsed]
module.vlan-map.ibm_network_vlan.managed_private["dal12"]: Still creating... [10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [10s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [10s elapsed]
module.vlan-map.ibm_network_vlan.managed_private["dal12"]: Creation complete after 12s [id=3199158]
module.vlan-map.ibm_network_vlan.managed_private["dal10"]: Creation complete after 12s [id=3199156]
module.vlan-map.ibm_network_vlan.managed_public["dal10"]: Creating...
module.vlan-map.ibm_network_vlan.managed_public["dal12"]: Creating...
module.cos-resource.ibm_resource_instance.cos-resource-instance: Still creating... [20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [20s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [20s elapsed]
module.cos-resource.ibm_resource_instance.cos-resource-instance: Creation complete after 22s [id=crn:v1:bluemix:public:cloud-object-storage:global:a/d778b72b327f4fa2ba986e3ee62fe017:62fe9ea6-21e8-42dc-8a9b-ca629d68280c::]
module.cos-resource.ibm_resource_key.cos_resource_key["b2c-auth-prod-cos-global-writer-key"]: Creating...
ibm_cos_bucket.prometheus: Creating...
ibm_cos_bucket.thanos: Creating...
ibm_cos_bucket.loki: Creating...
module.vlan-map.ibm_network_vlan.managed_public["dal12"]: Creation complete after 10s [id=3199160]
module.vlan-map.ibm_network_vlan.managed_public["dal10"]: Still creating... [10s elapsed]
module.vlan-map.ibm_network_vlan.managed_public["dal10"]: Creation complete after 11s [id=3199162]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Creating...
ibm_cos_bucket.thanos: Creation complete after 3s [id=crn:v1:bluemix:public:cloud-object-storage:global:a/d778b72b327f4fa2ba986e3ee62fe017:62fe9ea6-21e8-42dc-8a9b-ca629d68280c:bucket:auth-prod-thanos-us-south:meta:rl:us-south:public]
ibm_cos_bucket.loki: Creation complete after 3s [id=crn:v1:bluemix:public:cloud-object-storage:global:a/d778b72b327f4fa2ba986e3ee62fe017:62fe9ea6-21e8-42dc-8a9b-ca629d68280c:bucket:auth-prod-loki-dal-01:meta:rl:us-south:public]
module.cos-resource.ibm_resource_key.cos_resource_key["b2c-auth-prod-cos-global-writer-key"]: Creation complete after 5s [id=crn:v1:bluemix:public:cloud-object-storage:global:a/d778b72b327f4fa2ba986e3ee62fe017:62fe9ea6-21e8-42dc-8a9b-ca629d68280c:resource-key:a454022c-304a-43d0-9d42-c435ff9f5328]
ibm_cos_bucket.prometheus: Creation complete after 5s [id=crn:v1:bluemix:public:cloud-object-storage:global:a/d778b72b327f4fa2ba986e3ee62fe017:62fe9ea6-21e8-42dc-8a9b-ca629d68280c:bucket:auth-prod-prom-dal-01:meta:rl:us-south:public]
module.pg-umsn.ibm_database.database: Still creating... [30s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [40s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [50s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [30s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [1m0s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [1m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [1m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [40s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [1m10s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [1m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [1m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [50s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [1m20s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [1m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [1m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [1m0s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [1m30s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [1m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [1m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [1m10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [1m40s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [1m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [1m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [1m20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [1m50s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [1m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [1m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [1m30s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [2m0s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [2m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [2m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [1m40s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [2m10s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [2m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [2m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [1m50s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [2m20s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [2m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [2m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [2m0s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [2m30s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [2m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [2m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [2m10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [2m40s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [2m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [2m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [2m20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [2m50s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [2m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [2m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [2m30s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [3m0s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [3m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [3m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [2m40s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [3m10s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [3m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [3m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [2m50s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [3m20s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [3m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [3m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [3m0s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [3m30s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [3m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [3m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [3m10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [3m40s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [3m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [3m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [3m20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [3m50s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [3m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [3m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [3m30s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [4m0s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [4m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [4m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [3m40s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [4m10s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [4m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [4m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [3m50s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [4m20s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [4m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [4m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [4m0s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [4m30s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [4m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [4m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [4m10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [4m40s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [4m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [4m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [4m20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [4m50s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [4m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [4m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [4m30s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [5m0s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [5m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [5m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [4m40s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [5m10s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [5m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [5m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [4m50s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [5m20s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [5m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [5m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [5m0s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [5m30s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [5m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [5m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [5m10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [5m40s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [5m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [5m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [5m20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [5m50s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [5m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [5m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [5m30s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [6m0s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [6m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [6m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [5m40s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [6m10s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [6m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [6m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [5m50s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [6m20s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [6m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [6m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [6m0s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [6m30s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [6m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [6m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [6m10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [6m40s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [6m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [6m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [6m20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [6m50s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [6m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [6m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [6m30s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [7m0s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [7m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [7m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [6m40s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [7m10s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [7m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [7m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [6m50s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [7m20s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [7m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [7m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [7m0s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [7m30s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [7m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [7m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [7m10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [7m40s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [7m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [7m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [7m20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [7m50s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [7m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [7m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [7m30s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [8m0s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [8m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [8m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [7m40s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [8m10s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [8m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [8m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [7m50s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [8m20s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [8m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [8m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [8m0s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [8m30s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [8m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [8m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [8m10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [8m40s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [8m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [8m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [8m20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [8m50s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [8m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [8m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [8m30s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [9m0s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [9m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [9m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [8m40s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [9m10s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [9m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [9m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [8m50s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [9m20s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [9m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [9m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [9m0s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [9m30s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [9m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [9m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [9m10s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [9m40s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [9m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [9m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [9m20s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [9m50s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [9m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [9m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [9m30s elapsed]
module.pg-umsn.ibm_database.database: Still creating... [10m0s elapsed]
module.pg-umwu.ibm_database.database: Still creating... [10m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [10m0s elapsed]
module.pg-umwu.ibm_database.database: Creation complete after 10m2s [id=crn:v1:bluemix:public:databases-for-postgresql:us-south:a/d778b72b327f4fa2ba986e3ee62fe017:829f9b9e-5b6d-4b90-b0e3-a2ffe6a529d8::]
module.pg-umsn.ibm_database.database: Creation complete after 10m3s [id=crn:v1:bluemix:public:databases-for-postgresql:us-south:a/d778b72b327f4fa2ba986e3ee62fe017:5ef26b88-c220-4c48-8b24-371326452987::]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [9m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [10m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [9m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [10m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [10m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [10m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [10m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [10m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [10m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [10m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [10m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [11m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [10m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [11m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [10m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [11m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [11m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [11m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [11m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [11m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [11m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [11m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [11m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [12m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [11m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [12m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [11m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [12m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [12m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [12m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [12m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [12m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [12m20s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [12m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [12m30s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [13m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [12m40s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [13m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [12m50s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [13m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [13m0s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [13m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [13m10s elapsed]
module.pg-umbx.ibm_database.database: Still creating... [13m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [13m20s elapsed]
module.pg-umbx.ibm_database.database: Creation complete after 13m44s [id=crn:v1:bluemix:public:databases-for-postgresql:us-south:a/d778b72b327f4fa2ba986e3ee62fe017:f540b598-e90c-46f1-b847-9b2f51494637::]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [13m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [13m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [13m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [14m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [14m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [14m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [14m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [14m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [14m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [15m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [15m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [15m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [15m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [15m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [15m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [16m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [16m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [16m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [16m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [16m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [16m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [17m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [17m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [17m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [17m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [17m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [17m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [18m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [18m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [18m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [18m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [18m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [18m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [19m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [19m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [19m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [19m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [19m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [19m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [20m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [20m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [20m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [20m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [20m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [20m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [21m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [21m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [21m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [21m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [21m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [21m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [22m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [22m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [22m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [22m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [22m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [22m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [23m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [23m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [23m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [23m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [23m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [23m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [24m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [24m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [24m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [24m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [24m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [24m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [25m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [25m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [25m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [25m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [25m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [25m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [26m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [26m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [26m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [26m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [26m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [26m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [27m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [27m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [27m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [27m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [27m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [27m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [28m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [28m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [28m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [28m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [28m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [28m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [29m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [29m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [29m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [29m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [29m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [29m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [30m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [30m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [30m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [30m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [30m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [30m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [31m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [31m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [31m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [31m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [31m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [31m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [32m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [32m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [32m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [32m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [32m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [32m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [33m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [33m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [33m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [33m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [33m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [33m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [34m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [34m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [34m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [34m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [34m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [34m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [35m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [35m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [35m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [35m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [35m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [35m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [36m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [36m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [36m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [36m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [36m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [36m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [37m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [37m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [37m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [37m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [37m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [37m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [38m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [38m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [38m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [38m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [38m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [38m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [39m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [39m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [39m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [39m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [39m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [39m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [40m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [40m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [40m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [40m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [40m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [40m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [41m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [41m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [41m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [41m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [41m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [41m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [42m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [42m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [42m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [42m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [42m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [42m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [43m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [43m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [43m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [43m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [43m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [43m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [44m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [44m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [44m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [44m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [44m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [44m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [45m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [45m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [45m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [45m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [45m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [45m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [46m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [46m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [46m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [46m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [46m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [46m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [47m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [47m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [47m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [47m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [47m40s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [47m50s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [48m0s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [48m10s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [48m20s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Still creating... [48m30s elapsed]
module.cluster.ibm_container_cluster.cluster_autoscale[0]: Creation complete after 48m38s [id=c8f52pkd0v7ukku275bg]
data.ibm_container_cluster_config.cluster: Reading...
module.cluster.ibm_container_worker_pool.worker_pool["nlb"]: Creating...
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["accounting"]: Creating...
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Creating...
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Creating...
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Creating...
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["monitoring"]: Creating...
data.ibm_container_cluster_config.cluster: Read complete after 3s [id=c8f52pkd0v7ukku275bg]
module.cluster.ibm_container_worker_pool.worker_pool["nlb"]: Still creating... [10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["accounting"]: Still creating... [10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still creating... [10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Still creating... [10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Still creating... [10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["monitoring"]: Still creating... [10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Still creating... [20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still creating... [20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["nlb"]: Still creating... [20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["accounting"]: Still creating... [20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Still creating... [20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["monitoring"]: Still creating... [20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["accounting"]: Creation complete after 21s [id=b2c-auth-prod-iks-dal-01/c8f52pkd0v7ukku275bg-514a56c]
module.cluster.ibm_container_worker_pool.worker_pool["nlb"]: Still creating... [30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Still creating... [30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Still creating... [30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Still creating... [30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["monitoring"]: Still creating... [30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["alb"]: Creation complete after 31s [id=b2c-auth-prod-iks-dal-01/c8f52pkd0v7ukku275bg-c3b617f]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Still creating... [40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["nlb"]: Still creating... [40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Still creating... [40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["monitoring"]: Still creating... [40s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["monitoring"]: Creation complete after 46s [id=b2c-auth-prod-iks-dal-01/c8f52pkd0v7ukku275bg-4c28fac]
module.cluster.ibm_container_worker_pool.worker_pool["nlb"]: Still creating... [50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Still creating... [50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Still creating... [50s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["nlb"]: Still creating... [1m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Still creating... [1m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Still creating... [1m0s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Still creating... [1m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["nlb"]: Still creating... [1m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Still creating... [1m10s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["nlb"]: Still creating... [1m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Still creating... [1m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Still creating... [1m20s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"]: Creation complete after 1m20s [id=b2c-auth-prod-iks-dal-01/c8f52pkd0v7ukku275bg-2b62323]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Still creating... [1m30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["nlb"]: Still creating... [1m30s elapsed]
module.cluster.ibm_container_worker_pool.worker_pool["cassandra_operator"]: Creation complete after 1m33s [id=b2c-auth-prod-iks-dal-01/c8f52pkd0v7ukku275bg-3564cd8]
Warning: Empty provider configuration blocks are not required
on .terraform/modules/pg-umbx/modules/ibm/postgres/versions.tf line 11:
11: provider "ibm" {
Remove the ibm.leader_database_provider_alias provider block from
module.pg-umbx. Add ibm.leader_database_provider_alias to the list of
configuration_aliases for ibm in required_providers to define the provider
configuration name.
(and 2 more similar warnings elsewhere)
Error: Request failed with status code: 409, ServerErrorResponse: {"incidentID":"0d964d90-5c06-47ad-8c56-8bc67cb97ecd","code":"E0d39","description":"A worker pool with the same name already exists within the cluster. Choose another name.","type":"Provisioning"}
with module.cluster.ibm_container_worker_pool.worker_pool["nlb"],
on .terraform/modules/cluster/modules/ibm/iks-cluster/main.tf line 78, in resource "ibm_container_worker_pool" "worker_pool":
78: resource "ibm_container_worker_pool" "worker_pool" {
@hkantare Again same problem happen but this time with only one worker pool. Worker pool and worker nodes been created but terraform failed with error:
Error: Request failed with status code: 409, ServerErrorResponse: {"incidentID":"b5a25398-e92d-4e4f-b546-318640412868","code":"Ede7d","description":"The zone is already part of the worker pool.","type":"Provisioning","recoveryCLI":"Choose a different zone or a different worker pool to add the zone to."}
with module.cluster.ibm_container_worker_pool_zone_attachment.worker_pool_zone_attachment["nlb/dal12"],
on .terraform/modules/cluster/modules/ibm/iks-cluster/main.tf line 124, in resource "ibm_container_worker_pool_zone_attachment" "worker_pool_zone_attachment":
124: resource "ibm_container_worker_pool_zone_attachment" "worker_pool_zone_attachment" {
Error: Request failed with status code: 409, ServerErrorResponse: {"incidentID":"b5282076-32c8-4af6-839c-ae3a24fe73de","code":"Ede7d","description":"The zone is already part of the worker pool.","type":"Provisioning","recoveryCLI":"Choose a different zone or a different worker pool to add the zone to."}
with module.cluster.ibm_container_worker_pool_zone_attachment.worker_pool_zone_attachment["default/dal12"],
on .terraform/modules/cluster/modules/ibm/iks-cluster/main.tf line 124, in resource "ibm_container_worker_pool_zone_attachment" "worker_pool_zone_attachment":
124: resource "ibm_container_worker_pool_zone_attachment" "worker_pool_zone_attachment" {
Plan output: https://gist.github.com/DenysChekirda/e62696fc36cfe2e50f735d75a652f15f Apply output: https://gist.github.com/DenysChekirda/9e7383df80a4602e172f11bb83b5a7fb
From IBM cloud console I can see that worker pool nlb been created in both regions dal10, dal12 and each worker pool have worker nodes.
Hi @hkantare ,
I've been communicating with IBM Support team. By information from them they've already found the bug on Schematics and applied that to Prod but the actual issue is still exists.
And based on that my thoughts are like, do you have any possibilities to work with IBM team to figure out root cause of the issue? Could it be some kind of mishandled error which appears during terraform apply?
Thanks.
@DenysChekirda Its not the bug on Scheamtics...It was a fix from the IKS - API team. If you have recent logs may be can you open an issue against the kubernetes
This was the internal issue on IKS side https://github.ibm.com/alchemy-containers/armada-ironsides/issues/4275
@DenysChekirda Please can you provide any error messages and in particular incident IDs from your most recent occurrence?
I want to be sure that the bug referenced in your interactions with support would have covered attaching a zone to the worker pool.
Hi @lewiseevans here is the most recent error message with incident ID:
Error: Request failed with status code: 409, ServerErrorResponse: {"incidentID":"c2c5549c-39cf-4be3-9aae-c0ab62e63e37","code":"E0d39","description":"A worker pool with the same name already exists within the cluster. Choose another name.","type":"Provisioning"}
with module.cluster.ibm_container_worker_pool.worker_pool_autoscale["user_management"],
on .terraform/modules/cluster/modules/ibm/iks-cluster/main.tf line 99, in resource "ibm_container_worker_pool" "worker_pool_autoscale":
99: resource "ibm_container_worker_pool" "worker_pool_autoscale" {
@lewiseevans @hkantare We are seeing this ALOT in the past day or 2. It seems the initial response is returning a 500 error, and the retry is then getting a 409 conflict because the worker pool was actually created with the initial response (see below). I think the retry has to be removed for 500 errors unless the IKS team start to internally handle this in their code.
1st attempt:
[DEBUG] RESPONSE: [2024-01-31T22:19:19Z] Elapsed: 50232ms HTTP/1.1 500 Internal Server Error
Connection: close
Content-Length: 159
Cache-Control: max-age=0, no-cache, no-store
Content-Security-Policy: default-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; connect-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; font-src 'self'
Content-Type: application/json; charset=utf-8
Date: Wed, 31 Jan 2024 22:19:19 GMT
Expires: Wed, 31 Jan 2024 22:19:19 GMT
Pragma: no-cache
Server-Timing: cdn-cache; desc=MISS
Server-Timing: edge; dur=55
Server-Timing: origin; dur=50054
Server-Timing: ak_p; desc="1706739509812_389047309_600724259_5001518_3046_32_49_-";dur=1
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Carrier: prod-wdc04-carrier103
X-Content-Type-Options: nosniff
X-Correlation-Id: a992b6c9-a5f6-48f8-b150-d5ce4fadcd3a
X-Frame-Options: DENY
X-Hostname: armada-global-api-8678d99869-bzbq4
X-Permitted-Cross-Domain-Policies: master-only
X-Region: us-east
X-Request-Id: b9b7a6dd-70a5-4cec-a9d8-fa373b729e1a
X-Xss-Protection: 1; mode=block
{"incidentID":"b9b7a6dd-70a5-4cec-a9d8-fa373b729e1a","code":"G0002","description":"An internal error occurred while processing your request.","type":"General"}
Retry:
[DEBUG] RESPONSE: [2024-01-31T22:19:35Z] Elapsed: 10571ms HTTP/1.1 409 Conflict
Connection: close
Content-Length: 232
Cache-Control: max-age=0, no-cache, no-store
Content-Security-Policy: default-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; connect-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; font-src 'self'
Content-Type: application/json; charset=utf-8
Date: Wed, 31 Jan 2024 22:19:35 GMT
Expires: Wed, 31 Jan 2024 22:19:35 GMT
Pragma: no-cache
Server-Timing: cdn-cache; desc=MISS
Server-Timing: edge; dur=42
Server-Timing: origin; dur=10492
Server-Timing: ak_p; desc="1706739565011_389047309_601173084_1051430_3313_33_0_-";dur=1
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Carrier: prod-wdc04-carrier103
X-Content-Type-Options: nosniff
X-Correlation-Id: 558122f6-bd4e-42f0-8966-75f5189f170a
X-Frame-Options: DENY
X-Hostname: armada-global-api-8678d99869-hplp5
X-Permitted-Cross-Domain-Policies: master-only
X-Region: us-east
X-Request-Id: fb98a3c9-a061-40bb-bad9-957df8d642ac
X-Xss-Protection: 1; mode=block
{"incidentID":"fb98a3c9-a061-40bb-bad9-957df8d642ac,fb98a3c9-a061-40bb-bad9-957df8d642ac","code":"E0d39","description":"A worker pool with the same name already exists within the cluster. Choose another name.","type":"Provisioning"}
