TargetGroupBinding creation fails because ALBC unavailable during deployment

[igor-mendix]

Describe the bug When installing my ingress stack that includes AWS LB controller, I get errors similar to the following:

Error: failed to create resource: Internal error occurred: failed calling webhook "mtargetgroupbinding.elbv2.k8s.aws": failed to call webhook: Post "https://awslbc-webhook-service.ingress.svc:443/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding?timeout=10s": no endpoints available for service "awslbc-webhook-service"

If helm upgrade is run second time it completes fine.

Steps to reproduce Prepare a EKS cluster, create IAM role for ALBC (without ServiceAccount in k8s) and target group. Then deploy https://github.com/igor-mendix/ingress like this (beware of hardcoded ingress namespace):

helm upgrade -i ingress ./ -n ingress --create-namespace \
--set awslbc.clusterName=<your cluster name> \
--set awslbc.serviceAccount.create=true \
--set awslbc.serviceAccount.name=aws-load-balancer-controller \
--set awslbc.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn=<IAM role ARN> \
--set targetGroupARN=<target group ARN>

Expected outcome TargetGroupBinding created normally and in general no errors in deployment.

Environment

• Chart version: 1.4.2
• Kubernetes version: 1.22

Additional Context: I have a wrapper Helm chart linked above that has ALBC chart as a dependency. It deploys nginx ingress (which is also a dependency) behind ALB using additional Ingress resource for it. Creation of targets is up to ALBC based on this additional Ingress.

So it seems like some resources like TargetGroupBinding require ALBC to be fully operational at the time of creation. When I was attaching ALBC directly to ALB (letting it create its own TG), there was something similar, but it was not happening every time. Is there a way to make this work?

[kishorj]

@igor-mendix, TargetGroupBinding depends on the validating and mutating webhooks which is part of the LBC. The controller needs to be fully functional for the webhooks to be successful. In your case, it looks like the CRDs and the webhooks got installed but the controller pod is not yet functional.

Did you create the TargetGroupBinding manually or it was done by the controller for your Ingress resources?

[igor-mendix]

It is being created as part of the chart that I linked above. I don't know if there is a way to tell helm to wait for ALBC to become available before creating it, I'll research this.

[igor-mendix]

So I had to redo my deployment scheme to install ALBC chart separately, before TargetGroupBinding resource is created. My initial idea to have it as a dependency apparently wouldn't work because Helm just generates one huge template with all resources at once. Would still be nice if it was supported and TargetGroupBinding could be installed without ALBC pods running.

[kishorj]

@igor-mendix, the webhook provides validation and default values for the target group binding resources - this simplifies the controller implementation. The TargetGroupBinding resources are custom resources specific to this controller.

The controller needs to be up and running when you create/modify TGB resources. I'm closing the issue for now, if you have further concerns, feel free to reach out to us.