Unable to Expose Dashboard (hitting HTTP 404) on EKS with ELB and NLB

[imdhruva]

Hi there,

I have used the latest chart (version: 9.1.0) and enabled the following values

# Pod disruption budget
podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

# Create an IngressRoute for the dashboard
ingressRoute:
  dashboard:
    annotations:
      kubernetes.io/ingress.class: traefikv2

rollingUpdate:
  maxUnavailable: 1
  maxSurge: 1

globalArguments: []

ports:
  web:
    redirectTo: websecure

additionalArguments:
- "--providers.kubernetesingress.ingressclass=traefikv2"
- "--api.insecure=true"
- "--metrics.prometheus=true"

logs:
  general:
    level: DEBUG
  access:
    enabled: true

# Options for the main traefik service, where the entrypoints traffic comes
# from.
service:
  enabled: true
  type: LoadBalancer
  # Additional annotations (e.g. for cloud provider specific config)
  annotations:
    external-dns.alpha.kubernetes.io/hostname: traefikv2.test.<root-domain>
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <acm>
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"

rbac:
  namespaced: true

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app.kubernetes.io/instance
          operator: In
          values:
          - traefikv2
      topologyKey: kubernetes.io/hostname

hostNetwork: true

What we expected?

Dashboard should work on accessing url: http://traefikv2.test.<root-domain>/dashboard/

What we actually see?

HTTP 404 Error

curl -LI http://traefikv2.test.<root-domain>/dashboard/
HTTP/1.1 308 Permanent Redirect
Content-Length: 18
Content-Type: text/plain; charset=utf-8
Date: Wed, 26 Aug 2020 06:31:33 GMT
Location: https://traefikv2.test.<root-domain>/dashboard/
Connection: keep-alive

HTTP/1.1 404 Not Found
Content-Length: 19
Content-Type: text/plain; charset=utf-8
Date: Wed, 26 Aug 2020 06:31:34 GMT
X-Content-Type-Options: nosniff
Connection: keep-alive

Logs

10.13.110.13 - - [26/Aug/2020:06:31:33 +0000] "HEAD /dashboard/ HTTP/1.1" 308 18 "-" "-" 733 "web-to-443@internal" "-" 0ms
10.13.110.13 - - [26/Aug/2020:06:31:34 +0000] "HEAD /dashboard/ HTTP/1.1" - - "-" "-" 734 "-" "-" 0ms                     
10.13.109.117 - - [26/Aug/2020:06:31:41 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 735 "ping@internal" "-" 0ms

Helm Version

version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"dirty", GoVersion:"go1.14.3"}

K8s Version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-18T02:59:13Z", GoVersion:"go1.14.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-eks-2ba888", GitCommit:"2ba888155c7f8093a1bc06e3336333fbdb27b3da", GitTreeState:"clean", BuildDate:"2020-07-17T18:48:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

[imdhruva]

I tried the same thing with following changes:

• disabled ssl redirects
• disabled non-standard ingressClass name ie --providers.kubernetesingress.ingressclass=traefikv2
• added more logging information: - "--accesslog.fields.headers.names.RequestAddr=keep"
• updated dashboard ingressRoute to :

spec:
  entryPoints:
  - traefik
  routes:
  - kind: Rule
    match: Host(`traefik.test.<root-domain>`)
    services:
    - kind: TraefikService
      name: api@internal

I still only see the following: {"ClientAddr":"10.13.100.141:57655","ClientHost":"10.13.100.141","ClientPort":"57655","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":82601,"Overhead":82601,"RequestAddr":"traefik.test.<root-domain>","RequestContentSize":0,"RequestCount":767,"RequestHost":"traefik.test.<root-domain>","RequestMethod":"GET","RequestPath":"/","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"StartLocal":"2020-08-31T00:46:09.798277326Z","StartUTC":"2020-08-31T00:46:09.798277326Z","level":"info","msg":"","time":"2020-08-31T00:46:09Z"}{"ClientAddr":"10.13.100.141:57655","ClientHost":"10.13.100.141","ClientPort":"57655","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":100978,"Overhead":100978,"RequestAddr":"traefik.test.<root-domain>","RequestContentSize":0,"RequestCount":768,"RequestHost":"traefik.test.<root-domain>","RequestMethod":"GET" ,"RequestPath":"/favicon.ico","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"StartLocal":"2020-08-31T00:46:10.066274828Z","StartUTC":"2020-08-31T00:46:10.066274828Z","level":"info","msg":"","time":"2020-08-31T00:46:10Z"}

It still forwards all requests leading to 404 error. I do know this same pattern with ingress is working as-is in traefik1; traefik2 seems like a black box and hardly has any updates/responses.

[MRostanski]

Hi @imdhruva , are you certain that your traefik installation expects traefikv2 as ingress class? Default Helm installation uses traefik as a class name. Can you provide deployment arguments of traefik?

[imdhruva]

Hi @MRostanski , the reason why I have used traefikv2 as ingress class is that we have traefik as an ingress class already present and we want to upgrade traefik to v2. We use traefik as an ingress controller. The idea is we update the ingress annotations in all the affected apps to move them over from traefik v1 to traefik v2.

I am certain non-default ingressclass works. I have updated the the ingress annotations on the dependent apps and I can see them in traefikv2 ( I have to port-forward to check the dashboard) .

Can you provide deployment arguments of traefik?

I am not sure what you exactly mean by this since all the arguments we are overriding has already been provided above for traefikv2. There are no overrides we are providing during deployments. Over and above what's described here I am running this command to perform the deployment: helm install --debug traefikv2-test traefik/traefik -n test -f k8s-values.yaml (where k8s-values.yaml has the values described here ).

[SantoDE]

Im not too sure, if you can use the Dashboard like that with API insecure. Try without api insecure plx :)

[MRostanski]

@imdhruva Yes, I wes referring to any addidtional --set arguments, sorry if I was not clear. And also to be clear - I am not pointing any flaw in your issue, because I have the same issue as well, I solve it by deploying my own IngressRoute. That's the reason I visited this issue in the first place.

[MRostanski]

Also, when you look in the IngressRoute that is created, what helm chart does is the only entrypoint it has is traefik. Try to set it up as 'websecure`

[imdhruva]

@SantoDE I tried without - "--api.insecure=true" and I am very certain that it is required since not having that disables the entire dashboard - not even the port-forward from the pods work then. I even tried it with updated set of values (notice ports.traefik.expose: true):

# Pod disruption budget
podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

# Create an IngressRoute for the dashboard
ingressRoute:
  dashboard:
    annotations:
      kubernetes.io/ingress.class: traefikv2

globalArguments: []

ports:
  web:
    redirectTo: websecure
  traefik:
    expose: true

additionalArguments:
- "--api.insecure=true"
- "--metrics.prometheus=true"
- "--providers.kubernetesingress.ingressclass=traefikv2"
logs:
  general:
    format: json
  access:
    enabled: true
    fields:
      headers:
        defaultmode: keep
        names:
          RequestAddr: keep

# Options for the main traefik service, where the entrypoints traffic comes
# from.
service:
  # Additional annotations (e.g. for cloud provider specific config)
  annotations:
    external-dns.alpha.kubernetes.io/hostname: traefikv2.{{ .Environment.Name }}.{{ .Environment.Values.domain }}
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: Name={{ .Environment.Values.namespace }}-traefikv2,class=traefikv2,role={{ .Environment.Name }},vpc={{ .Environment.Values.kubectx }} 
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: {{ .Environment.Values.acm }}
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
rbac:
  namespaced: true

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app.kubernetes.io/instance
          operator: In
          values:
          - traefikv2
      topologyKey: kubernetes.io/hostname

hostNetwork: true

I created a new ingress-route:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubernetes.io/ingress.class: traefikv2
  labels:
    app.kubernetes.io/instance: traefikv2-test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-9.1.0
  name: traefikv2-test-dashboard-test
  namespace: test
spec:
  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(`traefikv2.test.<root-domain>`)
    priority: 10
    services:
    - kind: Service
      name: traefikv2-test
      namespace: test
      port: 9000

neither of these are working for me @MRostanski i tried websecure as well as traefik nothing is working.

I can though work it with the following ingress (with ports.traefik.expose: true):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
     kubernetes.io/ingress.class: traefikv2
  generation: 2
  name: traefikv2-test-dashboard
  namespace: test
spec:
  rules:
  - host: traefikv2.test.<root-domain>
    http:
      paths:
      - backend:
          serviceName: traefikv2-test
          servicePort: traefik
status:
  loadBalancer: {}

[SantoDE]

@imdhruva what is your wanted outcome? Like how do you want the dashboard to have accessible? From what I see, you want to have it accessible via http://traefikv2.test./dashboard/. For that to work, the best bet is to create an own IngressRoute which could look like this.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
spec:
  entryPoints:
    - web
  routes:
  - match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
    kind: Rule
    services:
    - name: api@internal
      kind: TraefikService

This is currently not supported by the helm-chart directly (at least not yet, see #186) but should work. With this IngressRoutein place, you should not need to have api.insecure

[imdhruva]

@SantoDE maybe I am doing something stupid - but I have tried this ingressRoute and it is not working for me/my configuration. I have tried this with entryPoint set as web as well as websecure. Again, I am not sure how you could get it to work with api.insecure not set since I do not see any dashboard even when I port-forward ports from traefik pods. Again the gist of my configs are:

• set NLB on
• set SSL on NLB
• force SSL redirection
• set ingressclass to ingressv2 I have tried these configs for ingressRoute:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: test
spec:
  entryPoints:
    - web
  routes:
  - match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
    kind: Rule
    services:
    - name: api@internal
      kind: TraefikService

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: test
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`traefikv2.test<root-domain) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
    kind: Rule
    services:
    - name: api@internal
      kind: TraefikService

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: test
spec:
  entryPoints:
    - websecure
  routes:
  - match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
    kind: Rule
    services:
    - name: api@internal
      kind: TraefikService

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: test
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`traefikv2.test<root-domain) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
    kind: Rule
    services:
    - name: api@internal
      kind: TraefikService

I still get 404 response. I continue to use k8s ingress with service set to expose traefik endpoint as a work-around

[gnarly-cl0s]

For the ingressroute,I would start with web endpoint to avoid TLS issues for starters and recommend you use the pathprefix since that should work (or at least does in my setup).

One thing that may be missing when the ingressroute is created from helm that I was looking into is the dashboard wouldn't work for me either unless port value is specified in services section. Try editing the ingressroute once it's created by including the expected port 9000 and see if that works.

I am not sure why this worked for me but hopefully it helps and can potentially be included as part of the dashboard-hook-ingressroute.yaml template as a default value if possible or makes sense.

[ncrmro]

I found adding a service and ingress helped

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: treafik
  name: treafik-dashboard
spec:
  ports:
    - name: dashboard
      port: 9000
      targetPort: 9000
  selector:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: treafik-dashboard-ingress
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`{{ traefik_dashboard_hostname }}`)
      kind: Rule
      services:
        - name: treafik-dashboard
          kind: Service
          port: 9000

[si458]

have you enabled tls on websecure? its disabled by default and with https the websecure needs tls! my config below works fine as basic example

logs:
  general:
    level: INFO
  access:
    enabled: true
ports:
  websecure:
    tls:
      enabled: true
additionalArguments:
  - --metrics.prometheus=true

[lifelofranco]

Were you ever able to solve this? Can't believe this simple problem had no resolution.

[vara-bonthu]

You don't need to deploy any Ingress objects if you install Traefik Ingress controller using this Helm Chart. It deploys a IngressRoute object for dashboard and it looks like below

kind: IngressRoute
apiVersion: traefik.containo.us/v1alpha1
metadata:
  annotations:
    helm.sh/hook: 'post-install,post-upgrade'
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-9.18.1
  name: traefik-dashboard
  namespace: kube-system
spec:
  entryPoints:
    - traefik
  routes:
    - kind: Rule
      match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
      services:
        - kind: TraefikService
          name: api@internal

You can use kubectl-portforward to test the WebUI

$ kubectl port-forward svc/traefik -n kube-system 9000:9000

Now open a local browser and enter http://localhost:9000/dashboard/

Remember to enter / at the end of dashboard prefix. You will get 404 without that

If you want to access the WebUI using NLB DNS or Route 53 then update the above IngressRoute object like below

  routes:
    - kind: Rule
      match: Host(`NLB DNS  or Route53 endpoint`) && PathPrefix(`/dashboard`) || PathPrefix(`/api`)

Now access using http://<hostname>:9000/dashboard/

[llavaud]

@vara-bonthu Traefik port (9000) is not exposed by default on the service so we must do the port-forward directly on the pod, but for me I still have a 404 response for the dashboard (with the ended /).

I have deployed the latest chart 9.18.2 with traefik 2.4.8.

[llavaud]

@vara-bonthu Traefik port (9000) is not exposed by default on the service so we must do the port-forward directly on the pod, but for me I still have a 404 response for the dashboard (with the ended /).

I have deployed the latest chart 9.18.2 with traefik 2.4.8.

Ok I just found out that we need to set --api.insecure=true to access the dashboard through a port-forward.

[alejandroEsc]

Any update on this. I am seeing very similar results and have tried multiple ingressroutes and flags as the original author of this ticket. This should be a fairly simple thing to do, not sure what could be going wrong .

[nscriptdotgo]

I am having a similar issue but for a non traefik service. I too have an ELB in front of Traefik, a service of type LoadBalancer and I am receiving 404s when trying to route to a particular url configured via IngressRoute, either http/https does not work. I do see an access log entry, but i am not getting routed to my API pod.

[jamiegwatkin]

I had the same problems as described above, I managed to expose the dashboard by doing the following:

Set values.yaml in helm to:

ingressRoute:
  dashboard:
    enabled: true
    # Additional ingressRoute annotations (e.g. for kubernetes.io/ingress.class)
    annotations: {}
    # Additional ingressRoute labels (e.g. for filtering IngressRoute by custom labels)
    labels: {}

I then created the following ingress route:

kind: IngressRoute
apiVersion: traefik.containo.us/v1alpha1
metadata:
  annotations:
    helm.sh/hook: 'post-install,post-upgrade'
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-9.18.1
  name: traefik-dashboard-test
  namespace: kube-system
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`hostname of alb`) && PathPrefix(`/dashboard`) || PathPrefix(`/api`)
      services:
        - kind: TraefikService
          name: api@internal

This goes with what @vara-bonthu was saying except note that the entrpoint is set to web and not traefik. This matches whats documented on the offical site - https://doc.traefik.io/traefik/getting-started/install-traefik/.

What's really infurating is that the ingress rotue created by the chart is wrong so if you just copy that with the added Host() match it doesn't work.

[mloiseleur]

I close this issue since now, there is some documentation on how to do it. It's hard to get it right, so don't hesitate to share how you think we can improve it.

[eastwoo]

seriously, I am deploying the latest traefik v2.9.6 on eks 1.23, still got 404. finally I switched to ingress-nginx controller. it is too hard for me to make traefik in the EKS cluster.

for this dashboard, if I am correct, I can use $lb_host/dashboard and access its dashboard, but I just got 404, I need to add / use $lb_host/dashboard/ to access its dashboard

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: dashboard
  namespace: traefik
spec:
  entryPoints:
    - web
  routes:
    - match: PathPrefix(`/dashboard`) || PathPrefix(`/api`, `/api/`)
      kind: Rule
      services:
        - name: api@internal
          kind: TraefikService

the ingress does not work either, does it means that we can not use the native ingress any more? only IngressRoute are supported?

[mloiseleur]

@eastwoo It's recommended to use IngressRoute as it's simpler to use.

If you want to keep using Ingress, maybe you will be interested by this detailed blog post on using Ingress with AWS.

There is also a detailed example on dashboard on this repo.

Does this works for you ?

For your question about using an Ingress with the dashboard, it works. In the linked PR, there is a complete example.