linkerd2
linkerd2 copied to clipboard
Default Server policy on linkerd-jaeger prohibits jaeger-ui access
What is the issue?
When deploying Jaeger extension using linkerd helm chart, it is not possible to have working Jaeger UI via Ingress due to Server policy on jaeger-ui
How can it be reproduced?
- Deploy linkerd and other components using Helm charts
helm upgrade --install linkerd-crds linkerd/linkerd-crds \
--namespace=linkerd \
--create-namespace
helm upgrade --install linkerd-cni linkerd/linkerd2-cni \
--namespace=linkerd \
--set destCNINetDir=/var/lib/rancher/k3s/agent/etc/cni/net.d \
--set destCNIBinDir=/var/lib/rancher/k3s/data/current/bin
# Because of race condition with CNI
sleep 10
helm upgrade --install linkerd-control-plane linkerd/linkerd-control-plane \
--namespace=linkerd \
--set-file identityTrustAnchorsPEM=ca.crt \
--set-file identity.issuer.tls.crtPEM=issuer.crt \
--set-file identity.issuer.tls.keyPEM=issuer-private.pem \
--set cniEnabled=true \
--set prometheusUrl="http://client-prometheus.monitoring.svc:9090" \
--set podMonitor.enabled=true
helm upgrade --install linkerd-viz linkerd/linkerd-viz \
--namespace=linkerd-viz \
--create-namespace \
--set enforcedHostRegexp=linkerd-viz-my-cluster.domain.com \
--set prometheusUrl="http://client-prometheus.monitoring.svc:9090" \
--set jaegerUrl="http://jaeger.linkerd-jaeger.svc:16686" \
--set prometheus.enabled=false
helm upgrade --install linkerd-jaeger linkerd/linkerd-jaeger \
--namespace=linkerd-jaeger \
--create-namespace
- Create a service in linkerd-viz namespace for jaeger ui
apiVersion: v1
kind: Service
metadata:
name: jaeger-external
namespace: linkerd-viz
spec:
externalName: jaeger.linkerd-jaeger
sessionAffinity: None
type: ExternalName
- Create an Ingress object to expose linkerd-viz and jaeger UIs
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/auth-realm: Authentication Required
nginx.ingress.kubernetes.io/auth-secret: web-ingress-auth
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Origin "";
proxy_hide_header l5d-remote-ip;
proxy_hide_header l5d-server-id;
nginx.ingress.kubernetes.io/upstream-vhost: $service_name.$namespace.svc.cluster.local:8084
name: ingress-route-linkerd-viz
namespace: linkerd-viz
spec:
ingressClassName: nginx
rules:
- host: linkerd-viz-my-cluster.domain.com
http:
paths:
- backend:
service:
name: web
port:
number: 8084
path: /
pathType: Prefix
- backend:
service:
name: jaeger-external
port:
number: 16686
path: /jaeger
pathType: Prefix
tls:
- secretName: domain.com
- Visit linkerd-viz-my-cluster.domain.com URL, see that linkerd-viz is working as expected.
- Open any jaeger trace for any component.
Expected result: Jaeger UI is working as expected
Actual result:
- Jaeger UI loads, but shows no data.
- After Jaeger pod restart UI is not loading and 403 error is seen in linkerd-proxy container of jaeger pod
Logs, error output, etc
[ 3776.356949s] INFO ThreadId(01) inbound:server{port=16686}: linkerd_app_inbound::policy::http: Request denied server.group=policy.linkerd.io server.kind=server server.name=jaeger-ui route.group= route.kind=default route.name=default client.tls=None(NoClientHello) client.ip=10.42.0.23
[ 3776.356999s] INFO ThreadId(01) inbound:server{port=16686}:rescue{client.addr=10.42.0.23:56060}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=client 10.42.0.23:56060: server: 10.42.0.41:16686: unauthorized request on route error.sources=[unauthorized request on route]
output of linkerd check -o short
For whatever reason linkerd check output is broken too
linkerd-viz
----------- \ Running viz extension check
‼ viz extension proxies are up-to-date
Get "https://versioncheck.linkerd.io/version.json?version=stable-2.14.10&uuid=unknown&source=cli": dial tcp: lookup versioncheck.linkerd.io on 10.26.3.12:53: no such host| Running viz extension check
see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cp-version for hints
ing viz extension check \ Running viz extension check
Status check results are × / Running viz extension check
user@win10-work:~$ extension check
ing viz extension check | Running viz extension check
nsion check - Running viz extension check
Running viz extension check
Environment
Kubernetes version: v1.28.8+k3s1 Cluster Environment: oVirt Host OS: Ubuntu 22.04 LTS Linkerd version: stable-2.14.10
Possible solution
Not a real solution, but I've found that deleting these two objects fixes access to jaeger ui
k delete AuthorizationPolicy jaeger-ui
k delete Server jaeger-ui
so it seems that they are missing some part for Ingress access, but I'm not sure which one would be that
Additional context
No response
Would you like to work on fixing this bug?
None
Yeah, that jaeger-ui AuthorizationPolicy is restricting access from viz' web
ServiceAccount only, used when using the jaeger site from the viz dashboard. To allow accessing via the ingress, you would need to add your ingress controller's ServiceAccount into that AuthorizationPolicy. The ingress controller would need to be meshed as well.