When using "CriticalAddonsOnly=true:NoSchedule" the AGIC "ingress-appgw-deployment" pod does not start

[philthynz]

trafficstars

I'm using the "only_critical_addons_enabled = true" in Terraform, which does the "CriticalAddonsOnly=true:NoSchedule". And when using this with the new "ingress_application_gateway" "enabled" = true option, the "ingress-appgw-deployment" pod does not start.

Code

resource "azurerm_kubernetes_cluster" "aks" {
  name                            = local.resource_name
  location                        = data.azurerm_resource_group.inf_resource_group.location
  resource_group_name             = azurerm_resource_group.k8s_resource_grp.name
  dns_prefix                      = local.resource_name
  node_resource_group             = local.node_resource_name
  kubernetes_version              = var.KUBERNETES.KubernetesVersion
  api_server_authorized_ip_ranges = var.KUBERNETES_ALLOWED_IP_RANGES
  tags                            = var.TAGS

  default_node_pool {
    name                         = "default"
    type                         = "VirtualMachineScaleSets"
    node_count                   = var.KUBERNETES.InitialNodeCount
    availability_zones           = var.KUBERNETES.AvailabilityZones
    vm_size                      = var.KUBERNETES.NodeVmSize
    enable_auto_scaling          = var.KUBERNETES.AutoScaleBool
    enable_host_encryption       = var.KUBERNETES.EnableHostEncryption
    os_disk_size_gb              = var.KUBERNETES.OSDiskSize
    os_disk_type                 = var.KUBERNETES.OSDiskType
    max_pods                     = var.KUBERNETES.MaxPods
    node_labels                  = var.KUBERNETES.Labels
    min_count                    = var.KUBERNETES.AutoScaleMinCount
    max_count                    = var.KUBERNETES.AutoScaleMaxCount
    vnet_subnet_id               = data.azurerm_subnet.aks_pods_subnet.id
    only_critical_addons_enabled = true
  }

  identity {
    type = "SystemAssigned"
  }

  role_based_access_control {
    enabled = true
    azure_active_directory {
      managed                = true
      admin_group_object_ids = var.KUBERNETES.AdminGroupObjectIds
    }
  }

  addon_profile {

    kube_dashboard {
      enabled = false
    }

    azure_policy {
      enabled = true
    }

    ingress_application_gateway {
      enabled    = true
      gateway_id = azurerm_application_gateway.app_gateway.id
    }
  }

  network_profile {
    load_balancer_sku = "standard"
    outbound_type     = "loadBalancer"
    network_plugin    = "azure"
    network_policy    = "azure"
  }
}

Errors

Looks like a bug.

[akshaysngupta]

thanks @philthynz for raising!

[wyp84]

I've a similar issue:

• When enabling the Ingress Application Gateway as an AKS Add-On (during our Bicep deployment):

addonProfiles: { ingressApplicationGateway: { enabled: true config: { applicationGatewayId: applicationGateway.id effectiveApplicationGatewayId: applicationGateway.id } }

...the Pod is scheduled on our User Node Pool, instead of the System Node Pool (which has the CriticalAddonsOnly=true:NoSchedule taint).

How can we force to create the Pod on our System Node Pool, when installing the Addon during the deployment (AKS managed)?

[yshehab]

@akshaysngupta I believe this is because the AGIC is classed as a "non-critical" addon. Is this likely to change?