`linkerd check` results in `tap API server has valid cert`

[aliabbasjaffri]

What is the issue?

We're creating linkerd and linkerd-viz via helm chart using terraform. linkerd and linkerd-viz get created successfully but when you run linkerd check to see the status of services, tap shows the following error:

$ linkerd check
Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2022-09-09T14:54:49Z
    see https://linkerd.io/2.11/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
‼ cli is up-to-date
    is running version 2.11.1 but the latest stable version is 2.12.0
    see https://linkerd.io/2.11/checks/#l5d-version-cli for hints

control-plane-version
---------------------
√ can retrieve the control plane version
‼ control plane is up-to-date
    is running version 2.11.1 but the latest stable version is 2.12.0
    see https://linkerd.io/2.11/checks/#l5d-version-control for hints
√ control plane and cli versions match

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
* linkerd-destination-7dbd867bbd-94p2f (stable-2.11.1)
* linkerd-identity-86cb476d86-6zfgx (stable-2.11.1)
* linkerd-proxy-injector-696b669dd7-jprm5 (stable-2.11.1)
    see https://linkerd.io/2.11/checks/#l5d-cp-proxy-version for hints
√ control plane proxies and cli versions match

Status check results are √

Linkerd extensions checks
=========================

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
× tap API server has valid cert
    cert is not issued by the trust anchor: x509: certificate signed by unknown authority
    see https://linkerd.io/2.11/checks/#l5d-tap-cert-valid for hints

Status check results are ×

Just to add a bit of context, we're creating the same secret in linkerd-viz namespace with the same ca key and pem which was passed to the linkerd issuer as a trust anchor. The error appears on linkerd dashboard as well, where any tap query leads to the following pop up appearing on the screen

An error has occurred.
Websocket close error [1011: Internal Error] : HTTP error, status Code [503] (unexpected API response: error trying to reach service: x509: certificate signed by unknown

How can it be reproduced?

• Install linkerd and linkerd-viz v2.11.1 via helm chart
• install linkerd cli tool for version 2.11.1
• run linkerd check and the error appears

Logs, error output, etc

Already pasted above

output of linkerd check -o short

$ linkerd check -o short
Linkerd core checks
===================

linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2022-09-09T14:54:49Z
    see https://linkerd.io/2.11/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints

linkerd-version
---------------
‼ cli is up-to-date
    is running version 2.11.1 but the latest stable version is 2.12.0
    see https://linkerd.io/2.11/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 2.11.1 but the latest stable version is 2.12.0
    see https://linkerd.io/2.11/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
* linkerd-destination-7dbd867bbd-94p2f (stable-2.11.1)
* linkerd-identity-86cb476d86-6zfgx (stable-2.11.1)
* linkerd-proxy-injector-696b669dd7-jprm5 (stable-2.11.1)
    see https://linkerd.io/2.11/checks/#l5d-cp-proxy-version for hints

Status check results are √

Linkerd extensions checks
=========================

linkerd-viz
-----------
× tap API server has valid cert
    cert is not issued by the trust anchor: x509: certificate signed by unknown authority
    see https://linkerd.io/2.11/checks/#l5d-tap-cert-valid for hints

Status check results are ×

Environment

• Kubernetes version: 1.21.9
• Cluster environment: AKS
• Host OS:
• Linkerd version: 2.11.1

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

maybe

[adleong]

Can you elaborate more on what you mean by:

Just to add a bit of context, we're creating the same secret in linkerd-viz namespace with the same ca key and pem which was passed to the linkerd issuer as a trust anchor.

By default, the Helm chart will generate TLS credentials and use these credentials in the tap-k8s-tls secret and the v1alpha1.tap.linkerd.io APIService (which need to match). It sounds like perhaps you manually replaced the tap-k8s-tls secret, with a certificate that the k8s API gateway cannot validate.

I would strongly recommend against creating this secret manually. Instead, either let the Helm chart automatically generate these resources, or else use cert-manager and the tap.injectCaFrom functionality.

[aliabbasjaffri]

Hello @adleong ! Thank you so much for the info. After following your recommendation, the error is solved! I believe this information is missing from the documentation:

Helm chart automatically generate these resources

which had us into thinking that we need to explicitly create those for ourselves. Would be awesome to have it documented so that others don't make the same mistake.

[adleong]

@aliabbasjaffri Awesome! We'd love a PR to our website repo https://github.com/linkerd/website/pulls if anything can be clarified.