pulumi-eks

pulumi-eks copied to clipboard

Hi! 👋 GCP has an add-on to enable their load balancer ingress controller directly from the API and console. It allows us to create an ingress using multiple paths and hosts, enable health checking, etc.

By default, the Kubernetes LoadBalancer service in EKS creates a classic load balancer, which lacks a lot of great features, like WebSockets and path mapping. It's also possible to enable the Network Load Balacing using a service annotation, but not the application load balancer.

AWS also has it's own ingress controller, which cannot be enabled in cluster creation, it requires some additional steps to be installed. A tutorial can be found here.

Since this package is all about simplifying the experience of using EKS and creating a load balancer is potentially a very common task, i believe this functionality could be added here, maybe behind a flag like enableAlbController, since it adds new resources to the cluster. 😅

Implementation Details

To enable the ALB ingress controller, it's necessary to:

1. Add a lot of IAM permissions to the node workers IAM role.
2. Add some RBAC roles and it's service account.
3. Add the ingress controller.
4. Create an ingress setting the annotation kubernetes.io/ingress.class to alb. More annotations can be found here.

Most of these steps can be done directly to the cluster through this Helm Chart or by adding it to the cluster using @pulumi/kubernetes, but i've struggled to implement the iam roles. @pulumi/eks already creates it's own instance role, so i'm not sure how i can implement a custom instance role without replacing the one created by the package or forking the package.

Alternatives

An alternative is to add a new option that allows the instance role to be customized, something like instanceRole to replace the default instance role created by the package or something like additionalInstanceRolePolicies to add more policies to the default instance role.

Note that the instanceRole is exposed from the component, so you should be able to attach additional RolePolicyAttachments to it after creating the cluster and before deploying the kubernetes resources.

That said, this may still be a good thing to offer an option for out of the box, as well as pointing to additional opportunities to expand the customization options on the component.

I just hit the need for this myself this morning, and it does seem like out of the box support for this would be really nice. I can imagine offering this as either a flag ok EKS cluster or as a separate resource that can be created separately if desired.

@metral @pgavlin -- this is an important feature not a blocker! Adding clarity based on suggestions from @lukehoban

Heres my reasoning on how this will help customers as they work with EKS.

1. AWS ALB has multiple features such as certs, WAF, HTTP/2, TLS offloading, Host/Path based routing, Cross zone LB that I have seen customers need as they scale their deployments.
2. Many of these features are not supported in NLB/ELB. Many of these features are harder for customers to work with in NGINX specifically since it is not maintained by the cloud provider.
3. SIG AWS has done alpha for Ingress groups that will allow users to work with smaller # of ALBs to better organize ingresses by namespaces.

Would be great to prioritize this feature support in M25/M26 to align with AWS roadmap and also recommend optimal ways of working with ALB to our customers. This will also allow us to have tighter engagement with the AWS LB team of fantastic engineers.

@metral -- I will push this to examples by next week so you can unassign yourself on this one.

Any updates on this? Is there a way to upload a controller and define an ingress using eks or do we need to supply the ALB controller yaml manifest files?

Cannot close issue without required labels: kind/, resolution/

Any updates on this please?

@jaxxstorm is this something that we can use the work in https://github.com/jaxxstorm/pulumi-aws-loadbalancercontroller/blob/main/nodejs/src/index.ts and bring this in as an out of the box adding that we can schematise and offer via this package? //cc @roothorp

Does Pulumi have a direct method for doing that, And is there a tutorial available for performing these steps with Pulumi ?

By default, the Kubernetes LoadBalancer service in EKS creates a classic load balancer, which lacks a lot of great features, like WebSockets and path mapping. It's also possible to enable the Network Load Balacing using a service annotation, but not the application load balancer.

AWS also has it's own ingress controller, which cannot be enabled in cluster creation, it requires some additional steps to be installed. A tutorial can be found here.

Since this package is all about simplifying the experience of using EKS and creating a load balancer is potentially a very common task, i believe this functionality could be added here, maybe behind a flag like enableAlbController, since it adds new resources to the cluster. sweat_smile