Make possible to add ingress object for each pod

[vitali-ipquants]

Hi Qdrant team,

Our team is building automation around Qdrant storage deployed with this helm chart. We're also using the snapshot APIs and for this we need to communicate with each pod directly. We found an elegant way to access the cluster API and each pod individually by using existing k8s facilities.

In short we:

• use cert-manager.io for issuing TLS certificats
• use Nginx Ingress which allows rewriting target urls
• we access cluster level API via https://<fqdn>/cluster
• we access each pod API via https://<fqdn>/pod-<index>

How we did it:

1. We use nginx ingress controller which allows to rewrite target url, we basically followed the answer from here. Our values contains this ingress definition:

ingress:
  annotations:
    cert-manager.io/cluster-issuer: production-cluster-issuer
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: "/$2"
    nginx.ingress.kubernetes.io/use-regex: 'true'
  enabled: true
  hosts:
    - host: qdrant-service.example.com
      paths:
        - path: "/cluster(/|$)(.*)"
          pathType: Prefix
          servicePort: 6333
  tls:
    - hosts:
        - qdrant-service.example.com
      secretName: qdrant-service.example.com

This ingress definition produces Ingress object pointing to the ClusterIP service which targets the three pods:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: production-cluster-issuer
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: qdrant
    meta.helm.sh/release-namespace: qdrant-service
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
....
spec:
  rules:
  - host: qdrant-service.example.com
    http:
      paths:
      - backend:
          service:
            name: qdrant-service
            port:
              number: 6333
        path: /cluster(/|$)(.*)
        pathType: Prefix
  tls:
  - hosts:
    - qdrant-service.example.com
    secretName: qdrant-service.example.com
status:
  loadBalancer:
    ingress:
    - hostname: qdrant-lb.example.com

2. After we deploy the helm chart, we additionally create ClusterIP service object and Ingress object for each pod.

• The ClusterIP service looks similarly to the one created by the helm chart, but in the selector section we add the pod-name label:

apiVersion: v1
kind: Service
....
  name: qdrant-service-0
  namespace: qdrant-service
spec:
....
  selector:
    app: qdrant
    app.kubernetes.io/instance: qdrant
    app.kubernetes.io/name: qdrant
    statefulset.kubernetes.io/pod-name: qdrant-service-0

• Then the backend of the per-pod Ingress object points to corresponding service (quadrant-service-0) and also has rewriting configuration for pod-0 path.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: production-cluster-issuer
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: "/$2"
    nginx.ingress.kubernetes.io/use-regex: 'true'
  labels:
    app: qdrant
    app.kubernetes.io/instance: qdrant
    app.kubernetes.io/name: qdrant
  name: qdrant-service-0
  namespace: qdrant-service
spec:
  rules:
    - host: qdrant-service.example.com
      http:
        paths:
          - backend:
              service:
                name: qdrant-service-0
                port:
                  number: 6333
            path: "/pod-0(/|$)(.*)"
            pathType: Prefix
  tls:
    - hosts:
        - qdrant-service.example.com
      secretName: qdrant-service.example.com  

Suggestion: It would be great if Qdrant helm chart supports to optionally define per pod Ingress objects. This would allow users like us to quite easily make each pod accessible.

Thank you!