Enhancement: Migrate ingress from nginx+loadbalancer service to true ingress

[InputObject2]

Is your feature request related to a problem? Please describe. Having nginx with a hardcoded config in a custom docker file is normally not a very kubernetes way of doing things.

Describe the solution you'd like In the charts, create the ingress configuration to match various paths.

Describe alternatives you've considered I've started working on this a bit and got it working as far as I can tell.

A configmap to hold the header configs:

custom-headers-configmap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-headers-configmap
  namespace: oneuptime
data:
  X-Real-IP: "$remote_addr"
  X-Forwarded-For: "$proxy_add_x_forwarded_for"
  X-Forwarded-Proto: "$scheme"
  Host: "$host"
  Upgrade: "$http_upgrade"
  Connection: "upgrade"

The main ingress that does no url rewrites (nginx-ingress requires different ingresses if you have different rewrite-targets):

main-ingress

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oneuptime-main-no-rewrites
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "256k"
    nginx.ingress.kubernetes.io/proxy-set-headers: "custom-headers-configmap"
    nginx.ingress.kubernetes.io/enable-websocket: "true"
    cert-manager.io/cluster-issuer: letsencrypt-production
    external-dns.alpha.kubernetes.io/hostname: oneuptime.example.com.
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - "oneuptime.example.com"
    secretName: tls-oneuptime
  rules:
  - host: "oneuptime.example.com"
    http:
      paths:
      - path: /
        pathType: Exact
        backend:
          service:
            name: oneuptime-status-page
            port:
              number: 3105
      - path: /status-page
        pathType: Prefix
        backend:
          service:
            name: oneuptime-status-page
            port:
              number: 3105
      - path: /accounts
        pathType: Prefix
        backend:
          service:
            name: oneuptime-accounts
            port:
              number: 3003
      - path: /otel-collector
        pathType: Prefix
        backend:
          service:
            name: oneuptime-otel-collector
            port:
              number: 4317
      - path: /ingestor
        pathType: Prefix
        backend:
          service:
            name: oneuptime-ingestor
            port:
              number: 3400
      - path: /dashboard
        pathType: Prefix
        backend:
          service:
            name: oneuptime-dashboard
            port:
              number: 3009
      - path: /admin
        pathType: Prefix
        backend:
          service:
            name: oneuptime-admin-dashboard
            port:
              number: 3158
      - path: /identity
        pathType: Prefix
        backend:
          service:
            name: oneuptime-identity
            port:
              number: 3087
      - path: /reference/
        pathType: Prefix
        backend:
          service:
            name: oneuptime-api-reference
            port:
              number: 1445
      - path: /file
        pathType: Prefix
        backend:
          service:
            name: oneuptime-file
            port:
              number: 3125
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: oneuptime-dashboard-api
            port:
              number: 3002
      - path: /realtime
        pathType: Prefix
        backend:
          service:
            name: oneuptime-dashboard-api
            port:
              number: 3002
      - path: /analytics-api
        pathType: Prefix
        backend:
          service:
            name: oneuptime-dashboard-api
            port:
              number: 3002
      - path: /workflow
        pathType: Prefix
        backend:
          service:
            name: oneuptime-workflow
            port:
              number: 3099
      - path: /l/
        pathType: Prefix
        backend:
          service:
            name: oneuptime-link-shortener
            port:
              number: 3521
      - path: /workers
        pathType: Prefix
        backend:
          service:
            name: oneuptime-workers
            port:
              number: 3452

And then 4 different ingresses for the different rewrite-targets we have:

status-page-api-ingress

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oneuptime-status-page-api
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /api/status-page/
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "256k"
    nginx.ingress.kubernetes.io/proxy-set-headers: "custom-headers-configmap"
    nginx.ingress.kubernetes.io/enable-websocket: "true"
    cert-manager.io/cluster-issuer: letsencrypt-production
    external-dns.alpha.kubernetes.io/hostname: oneuptime.example.com.
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - "oneuptime.example.com"
    secretName: tls-oneuptime
  rules:
  - host: "oneuptime.example.com"
    http:
      paths:
      - path: /status-page-api/
        pathType: Prefix
        backend:
          service:
            name: oneuptime-dashboard-api
            port:
              number: 3002

status-page-sso-ingress

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oneuptime-status-page-sso
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /status-page-sso/
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "256k"
    nginx.ingress.kubernetes.io/proxy-set-headers: "custom-headers-configmap"
    nginx.ingress.kubernetes.io/enable-websocket: "true"
    cert-manager.io/cluster-issuer: letsencrypt-production
    external-dns.alpha.kubernetes.io/hostname: oneuptime.example.com.
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - "oneuptime.example.com"
    secretName: tls-oneuptime
  rules:
  - host: "oneuptime.example.com"
    http:
      paths:
      - path: /status-page-sso-api/
        pathType: Prefix
        backend:
          service:
            name: oneuptime-identity
            port:
              number: 3087

status-page-identity

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oneuptime-status-page-identity
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /status-page/
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "256k"
    nginx.ingress.kubernetes.io/proxy-set-headers: "custom-headers-configmap"
    nginx.ingress.kubernetes.io/enable-websocket: "true"
    cert-manager.io/cluster-issuer: letsencrypt-production
    external-dns.alpha.kubernetes.io/hostname: oneuptime.example.com.
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - "oneuptime.example.com"
    secretName: tls-oneuptime
  rules:
  - host: "oneuptime.example.com"
    http:
      paths:
      - path: /status-page-identity-api/
        pathType: Prefix
        backend:
          service:
            name: oneuptime-identity
            port:
              number: 3087

ingestor

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oneuptime-ingestor
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /incoming-request/
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "256k"
    nginx.ingress.kubernetes.io/proxy-set-headers: "custom-headers-configmap"
    nginx.ingress.kubernetes.io/enable-websocket: "true"
    cert-manager.io/cluster-issuer: letsencrypt-production
    external-dns.alpha.kubernetes.io/hostname: oneuptime.example.com.
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - "oneuptime.example.com"
    secretName: tls-oneuptime
  rules:
  - host: "oneuptime.example.com"
    http:
      paths:
      - path: /heartbeat
        pathType: Prefix
        backend:
          service:
            name: oneuptime-ingestor
            port:
              number: 3400

Additional context I'd do the PR but I'm not sure where to put this in the charts and I'd rather ask first since maybe I'm missing some paths. All the buttons in the web UI work and the experience is butter-smooth as far as I can tell.