aws-load-balancer-controller
aws-load-balancer-controller copied to clipboard
ClientHello too large due to Go 1.23 post-quantum crypto – breaks with AWS Network Firewall (Suricata)
Bug Description After upgrading to AWS Load Balancer Controller v2.12.0 (which uses Go 1.23), the controller could not connect to AWS APIs when running behind AWS Network Firewall with Suricata inspection.
Root cause is Go 1.23’s support for post-quantum TLS (PQ TLS), which increases the size of the ClientHello message beyond a single packet. Suricata currently cannot handle multi-packet ClientHello messages, causing the firewall to silently drop the outbound TLS handshake.
This was also encountered by the Terraform team: :link: https://github.com/hashicorp/terraform-provider-aws/issues/39311
Suricata maintainers are tracking it here: :link: https://redmine.openinfosecfoundation.org/issues/7476
Steps to Reproduce
- Deploy AWS Load Balancer Controller v2.12.0 (or build it with Go 1.23).
- Place the controller behind AWS Network Firewall with Suricata rules enabled.
- Observe failed outbound TLS requests to AWS APIs.
Logs:
- TLS handshake fails silently (no meaningful logs unless inspected at packet level).
- Controller cannot authenticate with AWS or complete reconciliations.
Expected Behavior TLS handshake completes successfully. Controller communicates with AWS APIs as expected.
Actual Behavior The TLS handshake fails at the ClientHello stage because the packet is dropped due to its size.
- Impact: Controller cannot function; fails to manage resources.
- Frequency: Always, when behind AWS Network Firewall with Suricata.
Regression Was the functionality working correctly in a previous version ? Yes Yes. It works correctly in:
- AWS Load Balancer Controller v2.11.0 (using Go 1.22 or earlier)
Current Workarounds :white_check_mark: Downgrade to AWS Load Balancer Controller v2.11.0 :question: Or build with GODEBUG=pqschemes=0 to disable post-quantum TLS extensions.
Environment
- AWS Load Balancer controller version: v2.12.0
- Kubernetes version: 1.30
- Using EKS (yes/no), if so version?: yes
- Using Service or Ingress: Ingress
- AWS region: us-east-1
- How was the aws-load-balancer-controller installed:
- If helm was used then please show output of
helm ls -A | grep -i aws-load-balancer-controller - If helm was used then please show output of
helm -n <controllernamespace> get values <helmreleasename> - If helm was not used, then copy/paste the exact command used to install the controller, including flags and options.
- If helm was used then please show output of
- Current state of the Controller configuration:
kubectl -n <controllernamespace> describe deployment aws-load-balancer-controller
- Current state of the Ingress/Service configuration:
kubectl describe ingressclasseskubectl -n <appnamespace> describe ingress <ingressname>kubectl -n <appnamespace> describe svc <servicename>
Possible Solution (Optional) Provide a way to disable post-quantum crypto, either:
- Via env variable (e.g. GODEBUG=pqschemes=0),
- Or disable in Go build flags for now until Suricata adds support.
Contribution Intention (Optional)
- [ ] Yes, I'm willing to submit a PR to fix this issue
- [x] No, I cannot work on a PR at this time
Additional Context