Does anyone have a working AWS NLB configuration with ambassador?

[joost14]

Hi,

My apologies for asking this question in issues. Don't know where else to put it.

I am trying to setup a kubernetes cluster in AWS EKS that uses a NLB loadbalancer with ambassador edge stack/emissary ingress. However after hours of troubleshooting and trying I didn't get it to work.

Following this guide: https://www.getambassador.io/docs/edge-stack/1.14/topics/install/yaml-install/ Ambassador edge stack v1.14 installs just fine without any errors and creates all the necessary pods and resources. It also creates an ELB loadbalancer automatically (this one does nothing I think?). After following this: https://www.getambassador.io/docs/edge-stack/1.14/topics/running/ambassador-with-aws/ it sets it up a NLB in AWS. Everything seemed fine so far. However, after deploying a test application and pasting the public dns name of the NLB in a browser, the browser just keeps loading. And in the logs of ambassador we see 'http 200 (OK)', but no response in the browser. Seems like traffic gets in, but no traffic gets out? We don't have any firewalls, acl of security groups enabled that block any traffic. This test application worked fine when deployed on a prototype cluster in Virtualbox with MetalLB.

Could anyone share his/her config files for installing/configuring ambassador with NLB like the one below?

Important to note --> The config file in the guide should be changed to http and https? HTTP and HTTPS results in an error (capital letters not allowed).

apiVersion: v1
kind: Service
metadata:
  name: ambassador
  namespace: ambassador
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  ports:
  - name: HTTP --> http
    port: 80
    targetPort: 8080
  - name: HTTPS --> https
    port: 443
    targetPort: 8443
  selector:
    service: ambassador

Used software: Kubernetes 1.21 (in EKS) Ambassador edge stack 1.14

Test application used to test connection:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test
  labels:
    app: nginx-test
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: nginx-test
  template:
    metadata:
      labels:
        app: nginx-test
    spec:
      containers:
      - name: nginx-test
        image: nginx:alpine
        imagePullPolicy: Always
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-test
spec:
  selector:
    app: nginx-test
  ports:
    - protocol: TCP
      port: 80
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: nginx-test
spec:
  prefix: /
  rewrite: /
  service: nginx-test:80

[felix-dpg]

The problem is not in the AWS LB, read this https://github.com/emissary-ingress/emissary/issues/3894#issue-1034656091 Appear that ambassador guys dont read the issues in this repo. The issue persist at this moment, the worst is that exist only 1 images for emissary 2.X

[AlverezYari]

I've been fooling around trying to get this product work in our new Argo cluster for the better part of a day. They really need to go through their docs on AWS, its a mess and its a mismatch of instructions that have you creating multiple services that conflict etc and its about as clear as mud. Close to just moving to another product. EKS AWS should be smooth as butter install for a product like this.

[bernardolk]

I concur, this is terrible. I can't get ambassador + AWS NLB to work and we already tried an ALB configuration before also without success.

[d2k-klin]

did someone found a solution for this issue ? i was able to register ONLY one Host with Lets Encrypt, the following ones are have a problem and can't be reached for ACME to verify the request.....they are also not reachable through HTTP. Deleting the first register, and creating some of the following ones its working fine, but only for a single host.... Any idea ?

[AlverezYari]

My solution ultimately was to switch back to NGINX Ingress and configure it myself.

[cyrus-mc]

#4153 might be related.

I was able to get NLB working with the following setup

apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  labels:
    app.kubernetes.io/name: emissary
    app.kubernetes.io/instance: server
  name: https
  namespace: emissary
spec:
  hostBinding:
    namespace:
      from: SELF
  l7Depth: 1
  port: 8443
  # documentation says to use protocol over protocolStack but there is a current bug with that approach
  # https://github.com/emissary-ingress/emissary/issues/4153
  #protocol: HTTPSPROXY
  protocolStack: [ "PROXY", "TLS", "HTTP", "TCP" ]
  securityModel: XFP

---

apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  labels:
    app.kubernetes.io/name: emissary
    app.kubernetes.io/instance: server
  name: all
  namespace: emissary
spec:
  hostname: "*"
  tlsSecret:
    name: ambassador-certs
  tls:
    min_tls_version: v1.2
  mappingSelector:
    matchLabels:
      # associate all mappings that contain this label
      app.kubernetes.io/component: api

---

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
      #service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-attributes: "load_balancing.cross_zone.enabled=true"
  labels:
    app.kubernetes.io/component: ambassador-service
    app.kubernetes.io/instance: emissary-ingress
    app.kubernetes.io/managed-by: getambassador.io
    app.kubernetes.io/name: emissary-ingress
    app.kubernetes.io/part-of: emissary-ingress
    product: aes
  name: emissary-ingress
  namespace: emissary
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080
  - name: https
    port: 443
    targetPort: 8443
  selector:
    app.kubernetes.io/instance: emissary-ingress
    app.kubernetes.io/name: emissary-ingress
    profile: main
  type: LoadBalancer

The key for me is that the pre-canned protocol HTTPSPROXY didn't set the protocolStack correctly. So I had to over-ride what the default was.

[totallyben]

If it helps anyone at all, we had a problem with a vanilla helm install onto EKS with an NLB but it is now working fine.

The issue we were seeing was a Bad Request response when running a curl request to the NLB via http and on https we received a curl: (35) error:0A00010B:SSL routines::wrong version number.

Please note we have we have proxy protocol v2 enabled on the NLB to preserve the client IP address.

We actually had two issues in play:

1. We used the helm chart to install the default listeners which creates a listeners configured with protocol: HTTP and protocol: HTTPS, however, these listeners needed updating to use protocolStack: [ "PROXY", "HTTP", "TCP" ] and protocolStack: [ "PROXY", "TLS", "HTTP", "TCP" ] respectively, as mentioned in https://github.com/emissary-ingress/emissary/issues/4153. To do this we reinstalled via helm with createDefaultListeners: false and then created the Listeners manually. Here are the Listeners we have in place that are working:

apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: emissary-ingress-http-listener
  namespace: emissary
spec:
  hostBinding:
    namespace:
      from: ALL
  port: 8080
  protocolStack: [ "PROXY", "HTTP", "TCP" ]
  securityModel: XFP
---
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: emissary-ingress-https-listener
  namespace: emissary
spec:
  hostBinding:
    namespace:
      from: ALL
  port: 8443
  protocolStack: [ "PROXY", "TLS", "HTTP", "TCP" ]
  securityModel: XFP

2. Our Host definition was referencing a certififcate that was in another namespace so it couldn't be read. To fixed this we moved the Host definition into the same namepsace as the certificate. For reference we identified this via the following errors in the logs:

Secret xxxxx.crt.emissary unknown Host wildcard-host.emissary: invalid TLS secret xxxxxxg inactive

[ghostsquad]

I'm just trying to get NLB (Layer 4) with HTTP (no TLS) to work, and I'm gettin 400's.

module

  config:
    diagnostics:
      allow_non_local: true
      enabled: false
    use_proxy_proto: true

svc - attributes (I’m running the last aws-load-balancer-controller, so these annotations don’t necessarily match the ambassador docs, but they are correct, as I’ve verified at the NLB)

    service.beta.kubernetes.io/aws-load-balancer-attributes: access_logs.s3.bucket=,access_logs.s3.enabled=false,access_logs.s3.prefix=,deletion_protection.enabled=false,load_balancing.cross_zone.enabled=true
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal
    service.beta.kubernetes.io/aws-load-balancer-type: external

listener

spec:
  hostBinding:
    namespace:
      from: SELF
  port: 8080
  protocol: HTTP
  securityModel: INSECURE

host

spec:
  acmeProvider:
    authority: none
  hostname: '*'
  requestPolicy:
    insecure:
      action: Route
  selector:
    matchLabels:
      a8r.io/hostname: wildcard
      a8r.io/requestPolicy: route-insecure
  tlsSecret: {}

also verified the NLB has 3 healthy targets, and Proxy protocol v2: On

NLB Listener is listening on port 80, Target Group is set to forward to 8080 (the listening address of the pods, not the service)

---

Port forwarding to the service works, which definitely points to some issue between the NLB and the Ambassador Edge Pods

k port-forward svc/ambassador-edge-default 8080:80
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080

http localhost:8080
HTTP/1.1 200 OK
...

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

[ghostsquad]

I can confirm that the protocol stack is the problem. Removing protocol and setting protocolStack: [ "PROXY", "HTTP", "TCP" ] Fixes the issue for me.

# Bug in using `protocol`
# https://github.com/emissary-ingress/emissary/issues/3904
protocolStack: [ "PROXY", "HTTP", "TCP" ]

[Azrrael]

We had a similar issues with k8s v1.27 and an Emissary-Ingress kubernetes yaml install. We tried Emissary-Ingress v3.7.0 but got the following error:

Error from server: error when creating "https://app.getambassador.io/yaml/emissary/3.7.0/emissary-emissaryns.yaml": conversion webhook for getambassador.io/v3alpha1, Kind=Module failed: Post "https://emissary-apiext.emissary-system.svc:443/webhooks/crd-convert?timeout=30s": no endpoints available for service "emissary-apiext"

so we switched to v3.4.0 which we got already working on Azure AKS. After installing Emissary-Ingress 3.4.0 this way we deleted the LB created by default, and then applied:

apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: http-listener
  namespace: emissary
spec:
  port: 8080
  protocolStack: [ "PROXY", "HTTP", "TCP" ]
  securityModel: XFP
  hostBinding:
    namespace:
      from: SELF
---
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: https-listener
  namespace: emissary
spec:
  port: 8443
  protocolStack: [ "PROXY", "TLS", "HTTP", "TCP" ]
  securityModel: XFP
  hostBinding:
    namespace:
      from: SELF
  l7Depth: 1
---
#we had our cert configured as a secret in emissary's namespace
apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  labels:
    product: emissary
  name: our-cert
  namespace: emissary
spec:
  hostname: "*.our.cert"
  acmeProvider:
    authority: none
  requestPolicy:
    insecure:
      action: Redirect
      additionalPort: 8080
  tlsSecret:
    name: our-cert
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
  labels:
    app.kubernetes.io/component: ambassador-service
    app.kubernetes.io/instance: emissary-ingress
    app.kubernetes.io/managed-by: getambassador.io
    app.kubernetes.io/name: emissary-ingress
    app.kubernetes.io/part-of: emissary-ingress
    product: aes
  name: emissary-ingress
  namespace: emissary
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080
  - name: https
    port: 443
    targetPort: 8443
  selector:
    app.kubernetes.io/instance: emissary-ingress
    app.kubernetes.io/name: emissary-ingress
    profile: main
  type: LoadBalancer

which is a bit of a mishmash of the previous proposed configurations but it worked for us. Hope this helps anyone facing the same issue 🤞.