Default Server policy on linkerd-jaeger prohibits jaeger-ui access

[ValeriiVozniuk]

What is the issue?

When deploying Jaeger extension using linkerd helm chart, it is not possible to have working Jaeger UI via Ingress due to Server policy on jaeger-ui

How can it be reproduced?

1. Deploy linkerd and other components using Helm charts

helm upgrade --install linkerd-crds linkerd/linkerd-crds \
  --namespace=linkerd \
  --create-namespace
helm upgrade --install linkerd-cni linkerd/linkerd2-cni \
  --namespace=linkerd \
  --set destCNINetDir=/var/lib/rancher/k3s/agent/etc/cni/net.d \
  --set destCNIBinDir=/var/lib/rancher/k3s/data/current/bin
# Because of race condition with CNI
sleep 10
helm upgrade --install linkerd-control-plane linkerd/linkerd-control-plane \
  --namespace=linkerd \
  --set-file identityTrustAnchorsPEM=ca.crt \
  --set-file identity.issuer.tls.crtPEM=issuer.crt \
  --set-file identity.issuer.tls.keyPEM=issuer-private.pem \
  --set cniEnabled=true \
  --set prometheusUrl="http://client-prometheus.monitoring.svc:9090" \
  --set podMonitor.enabled=true
helm upgrade --install linkerd-viz linkerd/linkerd-viz \
  --namespace=linkerd-viz \
  --create-namespace \
  --set enforcedHostRegexp=linkerd-viz-my-cluster.domain.com \
  --set prometheusUrl="http://client-prometheus.monitoring.svc:9090" \
  --set jaegerUrl="http://jaeger.linkerd-jaeger.svc:16686" \
  --set prometheus.enabled=false
helm upgrade --install linkerd-jaeger linkerd/linkerd-jaeger \
  --namespace=linkerd-jaeger \
  --create-namespace

2. Create a service in linkerd-viz namespace for jaeger ui

apiVersion: v1
kind: Service
metadata:
  name: jaeger-external
  namespace: linkerd-viz
spec:
  externalName: jaeger.linkerd-jaeger
  sessionAffinity: None
  type: ExternalName

3. Create an Ingress object to expose linkerd-viz and jaeger UIs

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-realm: Authentication Required
    nginx.ingress.kubernetes.io/auth-secret: web-ingress-auth
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Origin "";
      proxy_hide_header l5d-remote-ip;
      proxy_hide_header l5d-server-id;
    nginx.ingress.kubernetes.io/upstream-vhost: $service_name.$namespace.svc.cluster.local:8084
  name: ingress-route-linkerd-viz
  namespace: linkerd-viz
spec:
  ingressClassName: nginx
  rules:
  - host: linkerd-viz-my-cluster.domain.com
    http:
      paths:
      - backend:
          service:
            name: web
            port:
              number: 8084
        path: /
        pathType: Prefix
      - backend:
          service:
            name: jaeger-external
            port:
              number: 16686
        path: /jaeger
        pathType: Prefix
  tls:
  - secretName: domain.com

4. Visit linkerd-viz-my-cluster.domain.com URL, see that linkerd-viz is working as expected.
5. Open any jaeger trace for any component.

Expected result: Jaeger UI is working as expected

Actual result:

1. Jaeger UI loads, but shows no data.
2. After Jaeger pod restart UI is not loading and 403 error is seen in linkerd-proxy container of jaeger pod

Logs, error output, etc

[  3776.356949s]  INFO ThreadId(01) inbound:server{port=16686}: linkerd_app_inbound::policy::http: Request denied server.group=policy.linkerd.io server.kind=server server.name=jaeger-ui route.group= route.kind=default route.name=default client.tls=None(NoClientHello) client.ip=10.42.0.23
[  3776.356999s]  INFO ThreadId(01) inbound:server{port=16686}:rescue{client.addr=10.42.0.23:56060}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=client 10.42.0.23:56060: server: 10.42.0.41:16686: unauthorized request on route error.sources=[unauthorized request on route]

output of linkerd check -o short

For whatever reason linkerd check output is broken too

                                                      linkerd-viz
-----------                               \ Running viz extension check
‼ viz extension proxies are up-to-date
    Get "https://versioncheck.linkerd.io/version.json?version=stable-2.14.10&uuid=unknown&source=cli": dial tcp: lookup versioncheck.linkerd.io on 10.26.3.12:53: no such host| Running viz extension check
    see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cp-version for hints
ing viz extension check                               \ Running viz extension check
Status check results are ×                / Running viz extension check
user@win10-work:~$  extension check
ing viz extension check                               | Running viz extension check
nsion check                               - Running viz extension check
  Running viz extension check

Environment

Kubernetes version: v1.28.8+k3s1 Cluster Environment: oVirt Host OS: Ubuntu 22.04 LTS Linkerd version: stable-2.14.10

Possible solution

Not a real solution, but I've found that deleting these two objects fixes access to jaeger ui

k delete  AuthorizationPolicy jaeger-ui
k delete Server jaeger-ui

so it seems that they are missing some part for Ingress access, but I'm not sure which one would be that

Additional context

No response

Would you like to work on fixing this bug?

None

[alpeb]

Yeah, that jaeger-ui AuthorizationPolicy is restricting access from viz' web ServiceAccount only, used when using the jaeger site from the viz dashboard. To allow accessing via the ingress, you would need to add your ingress controller's ServiceAccount into that AuthorizationPolicy. The ingress controller would need to be meshed as well.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.