Simplify e2e mTLS scenario for Ingress

[denniszielke]

Please describe the Improvement and/or Feature Request My primary use case for OSM is end-to-end mTLS. However it is a lot of effort to implement this with for example Nginx because I need not only to create an ingress object but also configs for IngressBackend, certificate and matching configs. This is hard to automate and introduces a lot of room for errors. I would like for OSM controller to generate these configs based on the existing configuration. It seems that everything needed is already there and OSM should be able to derive the osm-config, Ingressbackend and certificate AuthenticatedPrincipal just from the ingress annotations.

Scope (please mark with X where applicable)

• New Functionality [x]
• Install [ ]
• SMI Traffic Access Policy [ ]
• SMI Traffic Specs Policy [ ]
• SMI Traffic Split Policy [ ]
• Permissive Traffic Policy [ ]
• Ingress [x]
• Egress [ ]
• Envoy Control Plane [x]
• CLI Tool [ ]
• Metrics [ ]
• Certificate Management [ ]
• Sidecar Injection [ ]
• Logging [ ]
• Debugging [ ]
• Tests [ ]
• CI System [ ]
• Demo [ ]
• Project Release [ ]

Possible use cases I have a dozen endpoints that need to be created, updated and exposed with e2e mTLS and it is very tricky to automate this for every cluster in every environment. This could be a lot easier - especially in combination with the AKS web routing addon which is based on nginx. See https://github.com/Azure/AKS/issues/2489

[shashankram]

I would like for OSM controller to generate these configs based on the existing configuration. It seems that everything needed is already there and OSM should be able to derive the osm-config, Ingressbackend and certificate AuthenticatedPrincipal just from the ingress annotations.

@denniszielke thanks for bringing up your concern. Could you clarify what exactly are you expecting OSM to do differently in terms of autogenerating configs, and which configs you expect OSM to read to generate these configs?

OSM's ingress capability is meant to be decoupled from the ingress controller and API used to configure the ingress controller. Different ingress controllers use different Ingress APIs and annotations. It is not feasible for OSM to watch/parse different ingress controller annotations/APIs, and for this reason, the ingress configuration in OSM is decoupled from the Ingress controller configuration using the IngressBackend resource.

If you install OSM with Contour, then the certificate to use for mTLS between ingress pod and the backend pod is automated, see https://release-v1-1.docs.openservicemesh.io/docs/demos/ingress_contour/ for more info.

We want to give users the choice to use their own ingress controller, hence the decoupling of configuration is necessary.

[ams0]

@denniszielke I'd like to share here some instructions I came up with the invaluable help of @keithmattix for an mTLS scenario with ingress-nginx and keycloak, that might be of interest for you:

Let me go thru the setup :

1. Install a 1.24.0 cluster
2. Install OSM and add the namespaces:

osm install --set OpenServiceMesh.enablePermissiveTrafficPolicy=false
kubectl patch meshconfig osm-mesh-config -n osm-system -p '{"spec":{"traffic":{"enableEgress":true}}}'  --type=merge

kubectl create ns keycloak
osm namespace add keycloak
kubectl create ns ingress-nginx
osm namespace add keycloak

3. Install ingress-nginx, with fixes for this issue

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \
  --set controller.service.externalTrafficPolicy=Local \
  --namespace ingress-nginx --create-namespace

4. Add the public IP to DNS for resolution:

INGRESS_IP=`kubectl get svc -n ingress-nginx ingress-nginx-controller --output=jsonpath="{.status.loadBalancer.ingress[0]['ip']}"`
az network dns record-set a add-record  -n "*.osm" -g dns -z qubernetes.com --ipv4-addres $INGRESS_IP

5. Install cert-manager:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

6. Create a clusterIssuer, with specific annotations on the resolver pods:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
          podTemplate:
            metadata:
              annotations:
                "openservicemesh.io/inbound-port-exclusion-list": "8089"
EOF

7. Create the TrafficTarget and the refenced TCPRoute to allow traffic from keycloak to postgresql

kubectl apply -f - <<EOF
apiVersion: specs.smi-spec.io/v1alpha4
kind: TCPRoute
metadata:
  name: postgresql
  namespace: keycloak
spec:
  matches:
    ports:
    - 5432
---
apiVersion: access.smi-spec.io/v1alpha3
kind: TrafficTarget
metadata:
  name: postgresql
  namespace: keycloak
spec:
  destination:
    kind: ServiceAccount
    name: default
    namespace: keycloak
  rules:
  - kind: TCPRoute
    name: postgresql
  sources:
  - kind: ServiceAccount
    name: keycloak
    namespace: keycloak
EOF

8. Deploy keycloak with this values:

cat <<EOF > values.yaml
service:
  type: ClusterIP
ingress:
  enabled: true
  ingressClassName: "nginx"
  hostname: keycloak.osm.qubernetes.com
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_ssl_name "keycloak.keycloak.cluster.local";
    nginx.ingress.kubernetes.io/proxy-ssl-secret: osm-system/osm-nginx-client-cert
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
  tls: true
EOF

helm upgrade keycloak -i -n keycloak --create-namespace --values values.yaml bitnami/keycloak

9. Patch the keycloak-postgresql service with the appProtocol field; this won’t be necessary with the next release of OSM, but it is for the current (1.1.1):

kubectl patch svc -n keycloak keycloak-postgresql --type='json' -p='[{"op": "add","path": "/spec/ports/0/appProtocol","value": "TCP"}]'

10. Finally, the IngressBackend resource; note how this should list the POD port (8080) and not the service port for keycloak (80), note also that the protocol is https as it’s provided by the sidecar, while the app itself serves plain HTTP

kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: ingress
  namespace: keycloak
spec:
  backends:
  - name: keycloak
    port:
      number: 8080
      protocol: https
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    name: ingress-nginx-controller
    namespace: ingress-nginx
  - kind: AuthenticatedPrincipal
    name: ingress-nginx.ingress-nginx.cluster.local
EOF

This will setup ingress-nginx with let’s encrypt certificates terminated at the ingress and mTLS (the secret osm-system/ osm-nginx-client-cert) all the way to the keycloak pod from ingress pod and between keycloak and postgres (even if not explicitly set in neither).

[github-actions[bot]]

This issue will be closed due to a long period of inactivity. If you would like this issue to remain open then please comment or update.

[github-actions[bot]]

Issue closed due to inactivity.