Traefik crashes with Offloaded SSL + Google SSO loop

[JoeDurand3]

Hello, I'm having an issue with Auth Host + Selective Auth + Google SSO + Offloaded SSL.

First I was having an issue with a Google 400: redirect_uri mismatch caused by offloading my SSL to my AWS load balancer. So I added a middleware in front of traefik-forward-auth to force https with X-Forwarded-Proto = https. This got me to the login page, but maybe this compounds my problem, I'm not sure.

Next I was having the common issue of an infinite redirect loop mentioned in issues #158 and #149, so I added traefik-forward-auth as it's own middleware:

"traefik.http.routers.traefik-forward-auth.middlewares = traefik-forward-auth-headers,traefik-forward-auth@consulcatalog"

Now if I ping my test service, I see my traefik instance CPU spike through the roof where it promptly crashes with Exit Code: 137, Exit Message: "OOM Killed". I think what's happening is an infinite middleware loop that eats the CPU until it crashes.

Note that I'm using Nomad as my orchestrator. Very similar to k8s syntax, please let me know if i need to clarify something.

Here is my trefik + traefik-forward-auth config:

job "traefik" {
  region      = "us-west-1"
  datacenters = ["us-west-1a"]
  type        = "service"

  group "main" {
    count = 1

    restart {
      attempts = 1000 #for debugging
      delay    = "30s"
    }

    constraint {
      operator = "distinct_hosts"
      value    = "true"
    }

    network {
      port "http" {
        static = 80
      }

      port "api" {
        static = 8081
      }

      port "https" {
        static = 443
      }
    }

    service {
      name = "traefik"

      check {
        name     = "alive"
        type     = "tcp"
        port     = "http"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "instance" {
      driver = "docker"

      config {
        image        = "traefik:v2.3"
        network_mode = "host"

        volumes = [
          "local/traefik.yml:/etc/traefik/traefik.yml",
        ]
      }

      template {
        data        = file("./traefik-static-config.yml")
        destination = "local/traefik.yml"
      }

      template {
        data        = file("./traefik-adminui-config.yml")
        destination = "local/traefik/config/traefik-adminui-config.yml"
      }

      resources {
        cpu    = 100
        memory = 128
      }
    }

  }

  group "forward-auth" {

    network {
      port "http" {
        static = 4181
      }
    }

    task "instance" {
      driver = "docker"

      config {
        image = "thomseddon/traefik-forward-auth:2"
        ports = ["http"]
      }

      service {
        tags = [
          "traefik.enable=true",
          "traefik.http.routers.traefik-forward-auth.rule=Host(`oauth.<DOMAIN>`)",
          "traefik.http.routers.traefik-forward-auth.middlewares=traefik-forward-auth-headers,traefik-forward-auth@consulcatalog",
          "traefik.http.middlewares.traefik-forward-auth.forwardauth.address=https://oauth.<DOMAIN>",
          "traefik.http.middlewares.traefik-forward-auth.forwardauth.authResponseHeaders=X-Forwarded-User",
          "traefik.http.middlewares.traefik-forward-auth.forwardauth.trustForwardHeader=true",
          "traefik.http.middlewares.traefik-forward-auth-headers.headers.customrequestheaders.X-Forwarded-Proto=https",
          "traefik.http.services.traefik-forward-auth.loadbalancer.server.port=4181",
        ]
      }

      env {
        PROVIDERS_GOOGLE_CLIENT_ID     = "<CLIENT_ID>"
        PROVIDERS_GOOGLE_CLIENT_SECRET = "<CLIENT_SECRET>"
        SECRET = "<SECRET>"
        LOG_LEVEL = "trace"
        AUTH_HOST = "oauth.<DOMAIN>"
        COOKIE_DOMAIN = "<DOMAIN>"
      }
    }
  }

  group "test" {
    task "whoami" {
      driver = "docker"

      config {
        image = "containous/whoami"
      }

      service {
        tags = [
          "traefik.http.routers.whoami.rule=Host(`whoami.<DOMAIN>`)",
          "traefik.http.routers.whoami.middlewares=traefik-forward-auth"
        ]
      }
    }
  }
}

Thanks in advance.

[sureshamk]

Recently I came across a similar issue in Kubernetes. I added customRequestHeaders as middleware and its worked as expected.

Instead of traefik.http.middlewares.traefik-forward-auth-headers.headers.customrequestheaders.X-Forwarded-Proto=https

try traefik.http.middlewares.traefik-forward-auth-headers.headers.customRequestHeaders.X-Forwarded-Proto=https