assign static IP addresses to ingress controllers

[razvan-moj]

Background

Whenever we rebuild a controller, its NLBs are terminated and recreated with new IP addresses. This breaks connection from services that have a strict outgoing network policy

Approach

Add EIP allocations, 3 per controler (1 per zone)

Use the https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/service/annotations/#eip-allocations annotation to ensure new LBs come up with he same IPs

Definition of done

• [ ] ingress controllers module updated
• [ ] userguide updated to list the IPs
• [ ] another team member has reviewed
• [ ] smoke tests are green