[BUG] Ingress Template No Longer Works with Traefik

[dlford]

Describe the bug

The ingress template has changed since version 1.5.0 to use https instead of http. This results in Traefik responding with error 500 because the certificate of mailu-front is not valid for the host mailu-front.[namespace].svc.[cluster_domain].

Environment

• Kubernetes Platform: kubeadm cluster

Additional context

There are two possible solutions for this:

1. Revert the ingress back to using http instead of https
2. Add a Traefik ServersTransport for mailu-front with insecureSkipVerify: true, and reference it in the ingress template to bypass TLS verification for that host.

Option two will require some way of conditionally applying if Traefik is in use, of course.

EDIT to add: Option 3: make the http/https behavior an option in values.yaml (defaulting to https), this would be an easy solution.

[nextgens]

3. Make Traefik use proxy-protocol for https too, as documented upstream

[dlford]

@nextgens I'm not sure if/how that solves the issue?

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[migs35323]

i am also experiencing this issue, upgraded the chart from the version 1, to the latest one the ui\web interface is not working anymore, throwing 500

i have a traefik reverse proxy as ingress in kubernetes with all my other workloads

i did set up initially to have traefik being used for the web interface, but let the front pod deal with the mail related ports with a load balancer.

3. Make Traefik use proxy-protocol for https too, as documented upstream

@nextgens does this means i need to apply the insecure flag in traefik? https://mailu.io/master/reverse.html that is a big no.. are there better solutions?

[migs35323]

btw @dlford how about bring ya solution to this post? https://github.com/Mailu/helm-charts/issues/366#issuecomment-2336457641

[dlford]

To be clear, this is a temporary workaround for Traefik, this issue should really be resolved upstream.

1. Turn off the ingress in values.yaml (set enabled: false), it used to point to port 80 on mailu-front, but now points to port 443, which causes Traefik to throw an error 500 because the backend certificate is not valid for the domain [service].[namespace].svc.[cluster-domain]
2. Create a servers transport for mailu-front (make sure your cluster domain is cluster.local or change it in serverName below to match).

   apiVersion: traefik.io/v1alpha1
   kind: ServersTransport
   metadata:
     name: mailu-web
   spec:
     serverName: mailu-front.[namespace].svc.cluster.local
     insecureSkipVerify: true # <-- This fixes the issue
   

3. Create manually an ingress (and certificate if needed), but change the services to use the new servers transport.

   services:
     - name: mailu-front
       serversTransport: mailu-web
       port: 443
   

[migs35323]

thanks @dlford it worked!

I edited the front-pod service and added:

apiVersion: v1
kind: Service
metadata:
  name: mailu-front
  annotations:
    traefik.ingress.kubernetes.io/service.serverstransport: namespace-mailutransport@kubernetescrd

this made the web interface work right away, but i am wondering if there is a change to make it an open relay or something?

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[dlford]

@migs35323 Adding the servers-transport with insecureSkipVerify just tells Traefik to ignore the fact that the cert on mailu-front is not valid, it should not effect the mail services at all, only the Traefik proxy for webmail.

[nextgens]

Again, the right solution is to do what upstream documents.

Traefik should treat the traffic as TCP and forward it to mailu-front.

IMHO Ingress is the wrong abstraction, it should be a Gateway with a TCPRoute or a LoadBalancer (https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer)

[ArnCo]

I just setup the same workaround as @dlford. It makes no sense to make use of proxy-protocol for https, and it would also not fix the initial issue.

[nextgens]

I have yet to see someone propose a solution that works without. @dlford's clearly does not.

It's doable but tricky enough that most get it wrong; See #3684 for examples as of why.

The next major version of Mailu will most likely fingerprint clients, including their SSL stacks using JA3. At that point doing proxy-protocol will not be optional.

[dlford]

@nextgens you aren't being very clear...

the right solution is to do what upstream documents.

• where is this specifically documented? what is the solution? should I just search for "the right solution"?

I have yet to see someone propose a solution that works without.

• works without what exactly?

It's doable but tricky enough that most get it wrong

• what is doable? also, why should the helm chart create defaults that are "doable but tricky enough that most get it wrong"? Why make things intentionally difficult when they don't need to be?

IMHO Ingress is the wrong abstraction, it should be a Gateway with a TCPRoute or a LoadBalancer

• some of us run more than one domain/website and don't have multiple public IP addresses, this proposition is not compatible with that scenario.

At that point doing proxy-protocol will not be optional

• proxy-protocol is not well supported by all vendors (e.g. MikroTik), therefore requiring proxy-protocol is a poor decision IMO.

I get that proxy-protocol is great (when it actually works), but I don't think everyone wants this shoved down our throats with no alternative, especially considering how new it is and therefore not well documented or supported.

[dlford]

Ultimately, the main issue is that this helm chart does not work out of the box with Traefik. Of the many hundreds of helm charts I use or have used in the past, exactly one fails to work out of the box, and it's this one.

[nextgens]

proxy-protocol is not well supported by all vendors (e.g. MikroTik), therefore requiring proxy-protocol is a poor decision IMO

We don't make design decisions based on the lowest common denominator. Are you seriously suggesting this is what a tightly integrated tool like Mailu should do?

where is this specifically documented? what is the solution? should I just search for "the right solution"?

https://mailu.io/2024.06/reverse.html#traefik-as-reverse-proxy I don't know how we can make it any clearer; If you do PRs are welcome.

[dlford]

We don't make design decisions based on the lowest common denominator. Are you seriously suggesting this is what a tightly integrated tool like Mailu should do?

I am providing feedback and user opinion, this is clearly unwelcome, I will be seeking an alternative to Mailu and stopping my monthly donations.

Thanks for your "help".

[fastlorenzo]

I believe the issue is indeed the difference of behavior between ingress-nginx and traefik: ingress-nginx doesn't check the backend certificate by default (https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-certificate-authentication) but traefik does. For the sake of consistency, I propose to implement for now the option proposed by @dlford in order to disable backend certificate check with traefik (if traefik is being used as ingress). This would require a new option within the values.yaml like ingress.type: ingress-nginx or traefik

[fastlorenzo]

I made a first attempt at a fix in #390 but don't have a traefik setup to test it out. If someone is willing to test it and confirm it is working, I'll merge it.

[pgmillon]

I believe, this can be related to https://github.com/Mailu/helm-charts/issues/264

[NotANormalNerd]

Create a servers transport for mailu-front (make sure your cluster domain is cluster.local or change it in serverName below to match).

   apiVersion: traefik.io/v1alpha1
   kind: ServersTransport
   metadata:
     name: mailu-web
     namespace: mailu
   spec:
     serverName: mailu-front.mailu.svc.cluster.local
     insecureSkipVerify: true # <-- This fixes the issue

This is still a problem. I have found that instead of creating a new service you can just set the service annotations in the values.yaml

    front:
      service:
        annotations:
          traefik.ingress.kubernetes.io/service.serverstransport: mailu-mailu-web@kubernetescrd

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.