How to use EAS with Traefik IngressRoute CRD?

[thomafred]

trafficstars

Hi there!

We are looking to use EAS with an IngressRoute CRD.

This achievable, and if so, how?

[kettenbach-it]

You need a Middleware for this:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: my-middleware-name
  namespace: mynamespace
spec:
  forwardAuth:
    address: "https://myurl-to-eas.com/verify?config_token_store_id=default&config_token_id=whatever"
    trustForwardHeader: true
    authResponseHeaders:
      - X-Forwarded-User
      - X-Id-Token
      - X-Userinfo
      - X-Access-Token
      - Authorization

and use this in your IngressRoute:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: my-route
  namespace: mynamespace
spec:
  entryPoints:
    - https
  tls: {} # just use the wildcard certificate matching per domain
  routes:
  - match: Host(`my.host.com`) # Hostname to match
    kind: Rule
    middlewares:
      - name: my-middleware-name
        namespace: mynamespace
    services:
    - name: myservice
      port: 80

[travisghansen]

Been meaning to add a traefik 2 example. We’ll get one added to the docs.

[thomafred]

Thank you for your response. We have tested this, but are seeing some issues with how the redirect is handled.

From the logs, we are seeing the following (hostname redacted):

{"service":"external-auth-server","level":"verbose","message":"parent request info: {\"uri\":\"https://app.hostname.comundefined\",\"parsedUri\":{\"scheme\":\"https\",\"host\":\"eas.hostname.comundefined\",\"path\":\"\",\"reference\":\"absolute\"},\"parsedQuery\":{}}"}

Furthermore, the URL in the webbrowser also has the unhandled-part:

https://eas.hostname.comundefined/?__eas_oauth_handler__=authorization_callback&code=<AUTHORIZATION_CALLBACK>

[sseppola]

Ref. eas.hostname.comundefined, we found the undefined postfix to the domain comes from this line: https://github.com/travisghansen/external-auth-server/blob/master/src/utils.js#L191

It happens because req.headers["x-forwarded-uri"] was not defined nor checked for existence before use. We simply add the following to fix it:

originalRequestURI += req.headers["x-forwarded-uri"] || '';

[kettenbach-it]

Ref. eas.hostname.comundefined, we found the undefined postfix to the domain comes from this line: https://github.com/travisghansen/external-auth-server/blob/master/src/utils.js#L191

It happens because req.headers["x-forwarded-uri"] was not defined nor checked for existence before use. We simply add the following to fix it:

originalRequestURI += req.headers["x-forwarded-uri"] || '';

@travisghansen

I had this problem a while ago, too - before I set x-forwarded-uri - then it disappeared. I think this needs to be fixed in the code!

[travisghansen]

Agreed. Although in the pre reqs it is mentioned this is a must. Without setting it (to a proper value) there could be adverse effects.

Is traefik not sending it at all? Or simply not setting it when the path is the root path?

[sseppola]

From the logs we can't see any x-forwarded-uri header, so it was missing.

We tried to follow the HOWTO.md document as close as possible, but had to modify it to use the IngressRoute CRD

[travisghansen]

Interesting. Let me fire this up locally and do a little testing. What exact version of traefik are you running?

[sseppola]

Traefik 2.4.13

[travisghansen]

I think I may know what's going on here, but need more info to be sure.

In your config you have this: address: "https://myurl-to-eas.com/verify?config_token_store_id=default&config_token_id=whatever"

Is that going directly to eas using a k8s service or is that getting proxied through something (possibly even the same traefik instance as the original request)?

[thomafred]

Here is our values.yml that we use with the Helm-chart:

configTokenSignSecret: "${config_token_sign_secret}"
configTokenEncryptSecret: "${config_token_encrypt_secret}"
issuerSignSecret: "${issuer_sign_secret}"
issuerEncryptSecret: "${issuer_encrypt_secret}"
cookieSignSecret: "${cookie_sign_secret}"
cookieEncryptSecret: "${cookie_encrypt_secret}"
sessionEncryptSecret: "${session_encrypt_secret}"

logLevel: "silly"

redis-ha:
  enabled: false

 image:
   repository: hostname.azurecr.io/external-auth-server
   tag: "test-3"
   pullPolicy: IfNotPresent

ingress:
  enabled: false

We are using a Traefik IngressRoute for the ingress instead of the ingress included with the Helm-chart. As a result, the EAS-request is indeed getting proxied through the same Traefik instance as you suggest, however the IngressRoute has no middleware.

[travisghansen]

OK, is there any way you can point the middleware directly to the k8s internal service instead of through an Ingress/IngressRoute?

To give some context:

client -> traefik (actual service) -> traefik (eas)

I believe what's happening is the traefik in front of eas is actually stripping the headers (that get added by the traefik from the actual service).

You can bypass traefik fronting eas entirely (in the context of the /verify endpoint) OR you can probably add a proper forwardedHeaders value on the entryPoint (the entry point fronting eas).

• https://doc.traefik.io/traefik/routing/entrypoints/#forwarded-headers

Set the trustedIPs to the possible IPs of traefik itself...or set insecure to true. Obviously taking into consideration the security implications of your deployment.

In short X-Forwarded-Uri is a header specifically added to the forward auth endpoints and not generally added by traefik across the board. So an incoming request to an entryPoint with that value is likely getting stripped without further configuration to build 'trust' in the client.

[kettenbach-it]

Set the trustedIPs to the possible IPs of traefik itself...or set insecure to true. Obviously taking into consideration the security implications of your deployment.

What I forgot to mention: I had to set trustedIPs, to make it work, since my eas indeed is behind traefik.

[travisghansen]

For setting it as an forward auth url I would recommend using the internal service endpoint to avoid the overhead and issues.

However, I would also expose it externally using ingress/crd so that sso can be used by setting cookie domain and/or static callback url endpoint.

[thomafred]

Good evening,

Sorry about the late response - modifying the traefik config caused our SSL certificates to be renewed excessively, which in turn lead to us getting rate-limited by letsencrypt. Oops..

DEV-cluster only, so no real harm done :)

After solving the SSL-issue (also implementing persistance), we have added the changes you suggested. We added the AKS pod and service CIDR to trustedIPs.

Not sure why, but the Traefik middleware appear to want to redirect to port 8443. This is the default port of the websecure entrypoint in the Traefik Helm-chart.

[travisghansen]

Bad logic in eas or something else? Can you send over relevant logs?

[thomafred]

Seems I was tricked by the Firefox cache. Sorry about the confusion.

We were finally able to get the middleware to direct to the correct service (https://eas.myhost.com), and we are being prompted with the authentication flow, as expected.

Will let keep you updated as we progress

[thomafred]

The changes @travisghansen and @kettenbach-it suggested does in indeed work.

Quick summary:

First of all, we have deployed Traefik (version 2.4.13) in our case using the Helm-chart (version 10.1.2). Following is our values.yaml-file:

additionalArguments:
  - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
  - "--certificatesresolvers.letsencrypt.acme.email=${letsencrypt_email}"
  - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
  - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
  - "--api.insecure=true"
  - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
  - "--entryPoints.websecure.forwardedHeaders.trustedIPs=127.0.0.1,10.244.0.0/16,10.0.0.0/16"
  - "--serverstransport.insecureskipverify=true"

deployment:
  initContainers:
    # The "volume-permissions" init container is required if you run into permission issues.
    # Related issue: https://github.com/containous/traefik/issues/6972
    - name: volume-permissions
      image: busybox:1.31.1
      command: ["sh", "-c", "chmod -Rv 600 /data"]
      volumeMounts:
        - name: data
          mountPath: /data
logs:
  general:
    level: INFO
  access:
    enabled: true

persistence:
  enabled: true
  accessMode: ReadWriteMany
  storageClass: ssl-certificates

Do note that we have set --entryPoints.websecure.forwardedHeaders.trustedIPs=127.0.0.1,10.244.0.0/16,10.0.0.0/16, where 10.244.0.0/16 and 10.0.0.0/16 are the default pod and service CIDRs respectively for AKS.

We have also used Helm to deploy EAS. Following is our values.yaml file:

configTokenSignSecret: "${config_token_sign_secret}"
configTokenEncryptSecret: "${config_token_encrypt_secret}"
issuerSignSecret: "${issuer_sign_secret}"
issuerEncryptSecret: "${issuer_encrypt_secret}"
cookieSignSecret: "${cookie_sign_secret}"
cookieEncryptSecret: "${cookie_encrypt_secret}"
sessionEncryptSecret: "${session_encrypt_secret}"

redis-ha:
  enabled: false

ingress:
  enabled: false

Note that we have used a Traefik IngressRoute CRD to define the ingress instead of the ingress provided with the Helm-chart.

The middleware is basically the same as the one provided by @kettenbach-it:

**apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: my-middleware-name
  namespace: mynamespace
spec:
  forwardAuth:
    address: "https://eas.myhost.com/verify?config_token_store_id=default&config_token_id=whatever"
    trustForwardHeader: true
    authResponseHeaders:
      - X-Forwarded-User
      - X-Forwarded-Uri
      - X-Id-Token
      - X-Userinfo
      - X-Access-Token
      - Authorization

The same is also the case for the IngressRoute for the desired service:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: my-route
  namespace: mynamespace
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`app.myhost.com`) # Hostname to match
    kind: Rule
    middlewares:
      - name: my-middleware-name
        namespace: mynamespace
    services:
      - name: myservice
        port: 80
  tls:
    certResolver: letsencrypt
    domains:
      - app.myhost.com

[travisghansen]

Looks great!

Back to the original ask, I’ll probably add some code that throws an error instead of behaving badly as it does now.

[thomafred]

awesome, thank you!

@sseppola is also working on a PR for you

[travisghansen]

I have added more strict handling of this issue: https://github.com/travisghansen/external-auth-server/commit/b26a1a2ec62e0163d3a3cc24cbfd5bf4bdd0151b

If the necessary headers are not present it now throws an error.