EKS + AWS Network Load Balancer + Nginx? + TLS Termination

[taylorsmithgg]

Describe the bug I am using EKS in AWS

I have previously tested using a classic load balancer (ELB) in AWS, and using TLS termination with quepid works great.

However, due to some requirements with internal networking, we need to bind quepid to static ip addresses, which is an exclusive feature to Network Load Balancers in AWS.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=5
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    external-dns.alpha.kubernetes.io/hostname: quepid.mydomain.co
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:<aws-region>:<aws-account-id>:certificate/<acm-certificate-id>
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
    service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: 192.168.139.178, 192.168.97.111, 192.168.162.202
    kompose.cmd: kompose convert --file docker-compose.prod.yml
    kompose.version: 1.26.1 (HEAD)
  labels:
    io.kompose.service: app
  name: app
spec:
  type: LoadBalancer
  ports:
    - name: https
      port: 443
      targetPort: 3000
  selector:
    io.kompose.service: app
status:
  loadBalancer: {}

Using an NGINX sidecar w/the recommend nginx configuration, quepid is trying to redirect to http in logs:

quepid-prod-app 18:28:01 web.1    | I, [2023-07-17T18:28:01.270954 #13]  INFO -- : [edf58af4-11b0-47cc-817f-45b10fd98343] Redirected to http://quepid.mydomain.co/sessions/new

To Reproduce Steps to reproduce the behavior:

1. Convert docker compose prod build using kompose (requires some manual edits)
2. Deploy quepid in EKS
3. Create network load balancer using either port 3000 for direct connection or port 80 for NGINX

Expected behavior Ideally, there should be an easier way to do this with relative paths without requiring rewrites from NGINX. However, a fully working example/documentation would be sufficient.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context I have also tried toggling FORCE_SSL and the QUEPID_DOMAIN is set to https

Full configuration with NGINX sidecar:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=5
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    external-dns.alpha.kubernetes.io/hostname: quepid.mydomain.co
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:<aws-region>:<aws-account-id>:certificate/<acm-certificate-id>
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
    service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: 192.168.139.178, 192.168.97.111, 192.168.162.202
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    kompose.cmd: kompose convert --file docker-compose.prod.yml
    kompose.version: 1.26.1 (HEAD)
  labels:
    io.kompose.service: app
  name: quepid-nginx
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  ports:
    - name: https
      port: 443
      targetPort: 80
      protocol: TCP
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
  selector:
    io.kompose.service: app
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=5
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    external-dns.alpha.kubernetes.io/hostname: quepid-direct.mydomain.co
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:<aws-region>:<aws-account-id>:certificate/<acm-certificate-id>
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
    service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: 192.168.139.179, 192.168.97.112, 192.168.162.203
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    kompose.cmd: kompose convert --file docker-compose.prod.yml
    kompose.version: 1.26.1 (HEAD)
  labels:
    io.kompose.service: app
  name: quepid-direct
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  ports:
    - name: https
      port: 443
      targetPort: 3000
      protocol: TCP
    - name: http
      port: 80
      targetPort: 3000
      protocol: TCP
  selector:
    io.kompose.service: app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert --file docker-compose.prod.yml
    kompose.version: 1.26.1 (HEAD)
  labels:
    io.kompose.service: app
  name: app
spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: app
  strategy: {}
  template:
    metadata:
      annotations:
        kompose.cmd: kompose convert --file docker-compose.prod.yml
        kompose.version: 1.26.1 (HEAD)
      labels:
        io.kompose.service: app
    spec:
      volumes:
        - name: nginx-conf
          configMap:
            name: nginx-conf # place ConfigMap `nginx-conf` on /etc/nginx
            items:
              - key: nginx.conf
                path: nginx.conf
        - name: log
          emptyDir: {}
      containers:
        - image: nginx
          name: nginx
          ports:
            - containerPort: 80
          volumeMounts:
          - mountPath: /etc/nginx # mount nginx-conf volumn to /etc/nginx
            readOnly: true
            name: nginx-conf
          - mountPath: /var/log/nginx
            name: log

        - args:
            - foreman
            - s
            - -f
            - Procfile
          env:
            - name: GOOGLE_CLIENT_ID
              value: <google-client-id>
            - name: GOOGLE_CLIENT_SECRET
              value: <google-client-secret>
            - name: COMMUNAL_SCORERS_ONLY
              value: "false"
            - name: DATABASE_URL
              value: mysql2://root:password@mysql:3306/quepid
            - name: EMAIL_MARKETING_MODE
              value: "false"
            - name: EMAIL_PROVIDER
            - name: EMAIL_SENDER
            - name: FORCE_SSL
              value: "false"
            - name: MAX_THREADS
              value: "2"
            - name: PORT
              value: "3000"
            - name: PRIVACY_URL
            - name: QUEPID_DEFAULT_SCORER
              value: AP@10
            - name: QUEPID_DOMAIN
              value: https://quepid.mydomain.co
            - name: QUEPID_GA
            - name: QUERY_LIST_SORTABLE
              value: "true"
            - name: RACK_ENV
              value: production
            - name: RAILS_ENV
              value: production
            - name: RAILS_LOG_TO_STDOUT
              value: "true"
            - name: RAILS_SERVE_STATIC_FILES
              value: "true"
            - name: REDIS_URL
              value: redis://redis:6379/1
            - name: SECRET_KEY_BASE
              value: some_value_needed_here
            - name: SIGNUP_ENABLED
              value: "true"
            - name: TC_URL
            - name: WEB_CONCURRENCY
              value: "2"
          image: o19s/quepid:latest
          name: quepid-prod-app
          ports:
            - containerPort: 3000
          resources: {}
      restartPolicy: Always
status: {}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    events {
      worker_connections  10240;
    }

    http {
      server {
          listen              80;
          access_log off;

          location / {
              proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
              proxy_set_header X-Forwarded-Proto $scheme;
              proxy_set_header X-Forwarded-Host $host;
              proxy_pass http://localhost:3000;
          }
      }
    }

When pointed to port 3000 (no nginx) configuration does work on http but requires a manual swap to https once logged in.

when using FORCE_SSL it causes ERR_TOO_MANY_REDIRECTS

I cannot seem to get the login page on https

Latest update: the above configuration is working with nginx but breaks google auth because the redirect is http and not https

Error 400: redirect_uri_mismatch

Somewhat related, but different approach: https://github.com/o19s/quepid/issues/461