Tenant Console Ingress not working on Kubernetes with LVMs

[mihail-jipa]

Hello, i think i have discovered an issue with Min-io Tenant Console; i have deployed minio operator and tenant on a kubernetes cluster (1 controller + 3 workers); deployed using Helm and used 2 LVMs of 100G each on each worker node; i have created ingress for tenant with Helm, also added annotations how are presented on github docs, but i still get http 400 and when i am looking for logs, i find none when doing a request; however if i port forward the service of tenant console i can access it;

[cesnietor]

@mihail-jipa please share more details such as operator version. port-forward will work cause you are bypassing the ingress.

[mihail-jipa]

helm install --namespace minio-operator --create-namespace minio-operator operator-5.0.11.tgz --values operator-custom-values.yaml

helm install --namespace minio-tenant1 --create-namespace minio-tenant1 tenant-5.0.11.tgz --values tenant-custom-values.yaml

Here is the ingress generated by helm, also i added with comment ### the block ### added that could help, but did not

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: minio-tenant1
    meta.helm.sh/release-namespace: minio-tenant1
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: letsencrypt-minio
    #### also tried with and without next block
    #### https://github.com/minio/operator/blob/master/docs/nginx-ingress.md
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/server-snippet: |
      client_max_body_size 0;
    nginx.ingress.kubernetes.io/configuration-snippet: |
      chunked_transfer_encoding off;
    #### end of block
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: tenant1-console
  namespace: minio-tenant1
  resourceVersion: "19415169"
  uid: xxxxxxxxxxxxxxxx
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - tenant1.minio.mySite.com
      secretName: console-minio-tls-secret
  rules:
    - host: tenant1.minio.mySite.com
      http:
        paths:
          - backend:
              service:
                name: tenant1-console
                port:
                  name: https-console
            path: /
            pathType: Prefix
status:
  loadBalancer:
    ingress:
      - ip: xxxxxx
      - ip: xxxxxx
      - ip: xxxxxx
      - ip: xxxxxx

[jiuker]

I think you should set secretName with tenant-tls. Not console tls. @mihail-jipa console-minio-tls-secret is for minio-operator console . All you want is going to minio console from your rules.

[MaxThom]

Hi, I am facing a similar issue. I have installed the operator through the helm char successfully. I can access the operator console no issues. I can also install a tenant, but when I try to access the ingress, I get a 500 internal server error. I think it comes from traefik because using portfoward on the pod or service, it works.

I know its on the Traefik side, but I cant quite figure it out. The generated ingress seems right. Any idea?

OPERATOR

operator:
  console:
    ingress:
      enabled: true
      labels: {}
      annotations:
        cert-manager.io/cluster-issuer: ca-issuer
        cert-manager.io/common-name: operator.minio.*** #console.local
        cert-manager.io/duration: 2160h0m0s
        cert-manager.io/usages: "server auth"
        traefik.ingress.kubernetes.io/router.middlewares: networking-redirect-https@kubernetescrd
      host: operator.minio.***
      tls:
        - secretName: minio-tls-storage-staging
          hosts:
          - operator.minio.***
    securityContext:
      seccompProfile:
        type: RuntimeDefault
    containerSecurityContext:
      allowPrivilegeEscalation: false  
      capabilities:
            drop:
              - ALL
  operator:
    securityContext:  
      seccompProfile:
        type: RuntimeDefault
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
            drop:
              - ALL

TENANT

tenant:
  secrets:
    # To replace with a real secret
    name: myminio-env-configuration
    accessKey: minio 
    secretKey: minio123
  tenant:
    name: myminio
    pools:
      # For standalone mode, supply 1. For distributed mode, supply 4 or more.
      # Cant upgrade from 1 to >4. Must delete and recreate
      - servers: 1
        name: pool-myminio
        volumesPerServer: 4
        size: 10Gi
        annotations: {} # on pod TODO add prometheus annotations
        labels: {} # on pod
        nodeSelector: {}
        affinity: {}
        resources: {}
        securityContext:
          runAsUser: 1000
          runAsGroup: 1000
          fsGroup: 1000
          fsGroupChangePolicy: "OnRootMismatch"
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containerSecurityContext:
          allowPrivilegeEscalation: false
          capabilities:
                drop:
                  - ALL
    metrics:
      enabled: true
  ingress:
    api:
      enabled: true
      labels: {}
      annotations:
        cert-manager.io/cluster-issuer: ca-issuer
        cert-manager.io/common-name: myminio-api.minio.***
        cert-manager.io/duration: 2160h0m0s
        cert-manager.io/usages: "server auth"
        traefik.ingress.kubernetes.io/router.middlewares: networking-redirect-https@kubernetescrd
      tls:
        - secretName: myminio-api-minio-tls-storage
          hosts:
          - myminio-api.minio.***
      host: myminio-api.minio.***
    console:
      enabled: true
      labels: {}
      annotations:
        cert-manager.io/cluster-issuer: ca-issuer
        cert-manager.io/common-name: myminio.minio.***
        cert-manager.io/duration: 2160h0m0s
        cert-manager.io/usages: "server auth"
        traefik.ingress.kubernetes.io/router.middlewares: networking-redirect-https@kubernetescrd
      tls:
        - secretName: myminio-minio-tls-storage-apc-stating
          hosts:
          - myminio.minio..***
      host: myminio.minio.***

INGRESS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myminio-console
  annotations: 
    cert-manager.io/cluster-issuer: ca-issuer
    cert-manager.io/common-name: myminio.minio.***
    cert-manager.io/duration: 2160h0m0s
    cert-manager.io/usages: server auth
    traefik.ingress.kubernetes.io/router.middlewares: networking-redirect-https@kubernetescrd
spec:
  tls:
    - hosts:
        - "myminio.minio.***"
      secretName: myminio-minio-tls-storage
  rules:
    - host: myminio.minio.***
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myminio-console
                port:
                  name: https-console

[jiuker]

spec:
  tls:
    - hosts:
        - "myminio.minio.***"
      secretName: myminio-minio-tls-storage

Why do you set this secretName? @MaxThom

[MaxThom]

I have cert manager fetching a certificate and it create a tls secret with that name. Ive also been trying without all the TLS, and same issues. When looking at the Traefik dashboard, eveything seems ok.

[jiuker]

    cert-manager.io/usages: "server auth"
    traefik.ingress.kubernetes.io/router.middlewares: networking-redirect-https@kubernetescrd
  tls:
    - secretName: myminio-minio-tls-storage-apc-stating
      hosts:
      - myminio.minio..***
  host: myminio.minio.***

why set this with myminio.minio..*** ? @MaxThom

[MaxThom]

its not the actual value, but added *** just to hide the organization data. It is a proper dns that point to the correct ip that is set

[MaxThom]

ive set this value to false and I think its working now?

tenant:
  certificate:
    requestAutoCert: false

[jiuker]

@MaxThom You have .. at your DNS.

[MaxThom]

its just a typo when I remove it for github. My guess is that the TLS loaded on the ignress and the one in the tenant pod is not the same causing a 500 internal error.

[jiuker]

its just a typo when I remove it for github. My guess is that the TLS loaded on the ignress and the one in the tenant pod is not the same causing a 500 internal error.

You can get the pod Yaml and grep -tls found which secret mount the tls @MaxThom

[cesnietor]

closing due to innactivity. Please open a new one if the issue persists.