linkerd2 Linkerd stable-2.12.0 tap/top/route is not working

What is the issue?

Hi, @adleong

After installing Linkerd stable-2.12.0 version on GKE cluster 1.21.14-gke tap/top/routes components are not working in dashboard.

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd viz install | kubectl apply -f -
linkerd viz dashboard &

Note: I installed Linkerd stable-2.11.1 in a sample cluster everything was fine.

Screenshot 2022-09-16 at 12 21 00 installing

How can it be reproduced?

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd viz install | kubectl apply -f -
linkerd viz dashboard &

Logs, error output, etc

linkerd-destination

ubectl -n linkerd logs -f linkerd-destination-5b9d897fd9-d99d9                                                                                            ✔  allex-staging-next-k8s-cluster ⎈ 
Defaulted container "linkerd-proxy" out of: linkerd-proxy, destination, sp-validator, policy, linkerd-init (init)
[     0.005680s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.009519s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.009651s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.009656s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.009659s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.009690s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.009710s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.009735s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[     0.011725s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.018357s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN error.sources=[failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN, no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN]
[     0.021699s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN error.sources=[failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN, no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN]
[     0.114381s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.317085s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.727872s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.229032s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.729843s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.231777s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.732698s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.233701s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.734684s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     4.235685s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     4.736518s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     5.238359s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     5.740001s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     6.240847s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)

Linkerd tap

kubectl -n linkerd-viz logs -f tap-597c7b7c6b-jw7kl Defaulted container "linkerd-proxy" out of: linkerd-proxy, tap, linkerd-init (init)
[     0.004523s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.005598s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.005621s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.005625s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.005629s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.005632s]  INFO ThreadId(01) linkerd2_proxy: Local identity is tap.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
[     0.005636s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.005640s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.026771s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=tap.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
[     0.824078s]  INFO ThreadId(01) inbound:server{port=9998}:rescue{client.addr=10.183.6.1:54776}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[    88.021977s]  INFO ThreadId(01) inbound:server{port=9998}:rescue{client.addr=10.183.6.1:56272}: linkerd_app_core::errors::respond: Request failed error=connection closed before message completed
[    88.022045s]  INFO ThreadId(01) inbound:server{port=9998}:rescue{client.addr=10.183.6.1:56272}: linkerd_app_core::errors::respond: Request failed error=connection closed before message completed
[    88.022054s]  WARN ThreadId(01) inbound:server{port=9998}:rescue{client.addr=10.183.6.1:56272}: linkerd_app_inbound::http::server: Unexpected error error=connection closed before message completed
[    88.022186s]  INFO ThreadId(01) inbound:server{port=9998}:rescue{client.addr=10.183.6.1:56268}: linkerd_app_core::errors::respond: Request failed error=connection closed before message completed
[    88.022204s]  INFO ThreadId(01) inbound:server{port=9998}:rescue{client.addr=10.183.6.1:56268}: linkerd_app_core::errors::respond: Request failed error=connection closed before message completed
[    88.022209s]  WARN ThreadId(01) inbound:server{port=9998}:rescue{client.addr=10.183.6.1:56268}: linkerd_app_inbound::http::server: Unexpected error error=connection closed before message completed

Linkerd Web

kubectl -n linkerd-viz logs -f web-6485f66c86-qqn9h                                                                                                        ✔  allex-staging-next-k8s-cluster ⎈ 
Defaulted container "linkerd-proxy" out of: linkerd-proxy, web, linkerd-init (init)
[     0.080443s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.270233s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.271232s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.271314s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.271322s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.271326s]  INFO ThreadId(01) linkerd2_proxy: Local identity is web.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
[     0.271334s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.271339s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.290029s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=web.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
[    47.335297s]  INFO ThreadId(01) inbound:server{port=9994}:rescue{client.addr=10.183.3.1:49306}: linkerd_app_core::errors::respond: Request failed error=connection error: client: Connection reset by peer (os error 104) error.sources=[client: Connection reset by peer (os error 104)]
[   107.334788s]  INFO ThreadId(01) inbound:server{port=9994}:rescue{client.addr=10.183.3.1:50322}: linkerd_app_core::errors::respond: Request failed error=connection closed before message completed
[   107.334834s]  INFO ThreadId(01) inbound:server{port=9994}:rescue{client.addr=10.183.3.1:50322}: linkerd_app_core::errors::respond: Request failed error=connection closed before message completed
[   107.334844s]  WARN ThreadId(01) inbound:server{port=9994}:rescue{client.addr=10.183.3.1:50322}: linkerd_app_inbound::http::server: Unexpected error error=connection closed before message completed
[   117.334850s]  INFO ThreadId(01) inbound:server{port=9994}:rescue{client.addr=10.183.3.1:50498}: linkerd_app_core::errors::respond: Request failed error=connection error: client: Connection reset by peer (os error 104) error.sources=[client: Connection reset by peer (os error 104)]
[   137.335298s]  INFO ThreadId(01) inbound:server{port=9994}:rescue{client.addr=10.183.3.1:50844}: linkerd_app_core::errors::respond: Request failed error=connection error: client: Connection reset by peer (os error 104) error.sources=[client: Connection reset by peer (os error 104)]

output of `linkerd check -o short`

linkerd check -o short 
Linkerd extensions checks
=========================

linkerd-viz
-----------
‼ linkerd-viz ClusterRoles exist
    missing ClusterRoles: linkerd-default-metrics-api, linkerd-default-tap, linkerd-default-tap-admin
    see https://linkerd.io/2.12/checks/#l5d-viz-cr-exists for hints

Linkerd extensions checks
=========================

linkerd-viz
-----------
‼ linkerd-viz ClusterRoles exist
    missing ClusterRoles: linkerd-default-metrics-api, linkerd-default-tap, linkerd-default-tap-admin
    see https://linkerd.io/2.12/checks/#l5d-viz-cr-exists for hints

Status check results are

kubectl get clusterroles | grep linkerd-viz                                                                                                         
linkerd-linkerd-viz-metrics-api                                        2022-09-16T10:17:36Z
linkerd-linkerd-viz-prometheus                                         2022-09-16T10:17:37Z
linkerd-linkerd-viz-tap                                                2022-09-16T10:17:37Z
linkerd-linkerd-viz-tap-admin                                          2022-09-16T10:17:37Z
linkerd-linkerd-viz-web-api                                            2022-09-16T10:17:39Z
linkerd-linkerd-viz-web-check                                          2022-09-16T10:17:39Z

I had the same output on Linkerd stable-2.11.1

Environment

GKE cluster 1.21.14-gke

Linkerd: Client version: stable-2.12.0 Server version: stable-2.12.0

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

No response

Sep 16 '22 10:09 rohammosalli

+1, here, got the same problem on a fresh installation of a kops-managed k8s 1.21.14 and linkerd stable-2.12.0

Sep 18 '22 21:09 stanssh

Same problem for me while doing a trial of linkerd. I followed the exact steps mentioned in the getting started guide and then tried to follow the debug emojivoto guide but an empty white screen appears when clicking "tap" (and also verified that the same happens when I click "top")

Sep 20 '22 10:09 ttsakpc

+1 same issue here - and that's with 2.12.1

Sep 25 '22 18:09 winterhalder

Same problem for us, fresh install 1.21.0 from helm charts. First try on linkerd, unfortunately not working.

Sep 26 '22 06:09 ghost

We are facing the same issue. We installed a fresh stable-2.12.1 version for linkerd-viz and doing an port-forward to access the dashboard. Everything works fine except the tap, top, and route paths.