Linkerd pods fail to start up

[dreinhardt89]

What is the issue?

Did a helm release via terraform in a newly created cluster. After releasing the pods fail to startup.

How can it be reproduced?

With cert-manager installed perform a helm_release with identityTrustAnchorsPEM set to a tls self signed cert and cni enabled.

Logs, error output, etc

{"timestamp":"[     0.000595s]","level":"INFO","fields":{"message":"Using single-threaded proxy runtime"},"target":"linkerd2_proxy::rt","threadId":"ThreadId(1)"}
{"timestamp":"[     0.001090s]","level":"INFO","fields":{"message":"Admin interface on 0.0.0.0:4191"},"target":"linkerd2_proxy","threadId":"ThreadId(1)"}
{"timestamp":"[     0.001111s]","level":"INFO","fields":{"message":"Inbound interface on 0.0.0.0:4143"},"target":"linkerd2_proxy","threadId":"ThreadId(1)"}
{"timestamp":"[     0.001116s]","level":"INFO","fields":{"message":"Outbound interface on 127.0.0.1:4140"},"target":"linkerd2_proxy","threadId":"ThreadId(1)"}
{"timestamp":"[     0.001120s]","level":"INFO","fields":{"message":"Tap DISABLED"},"target":"linkerd2_proxy","threadId":"ThreadId(1)"}
{"timestamp":"[     0.001125s]","level":"INFO","fields":{"message":"Local identity is linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local"},"target":"linkerd2_proxy","threadId":"ThreadId(1)"}
{"timestamp":"[     0.001129s]","level":"INFO","fields":{"message":"Identity verified via localhost:8080"},"target":"linkerd2_proxy","threadId":"ThreadId(1)"}
{"timestamp":"[     0.001133s]","level":"INFO","fields":{"message":"Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)"},"target":"linkerd2_proxy","threadId":"ThreadId(1)"}
{"timestamp":"[     0.002353s]","level":"WARN","fields":{"message":"Failed to resolve control-plane component","error":"no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN"},"target":"linkerd_app_core::control","spans":[{"name":"policy"},{"port":"9990","name":"watch"},{"addr":"linkerd-policy.linkerd.svc.cluster.local:8090","name":"controller"}],"threadId":"ThreadId(1)"}
{"timestamp":"[     0.003756s]","level":"WARN","fields":{"message":"Failed to resolve control-plane component","error":"no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN"},"target":"linkerd_app_core::control","spans":[{"name":"policy"},{"port":"9990","name":"watch"},{"addr":"linkerd-policy.linkerd.svc.cluster.local:8090","name":"controller"}],"threadId":"ThreadId(1)"}
{"timestamp":"[     0.013593s]","level":"ERROR","fields":{"message":"Failed to certify identity: status: Internal, message: \"x509: requested SignatureAlgorithm does not match private key type\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\"} }"},"target":"linkerd_proxy_identity::certify","spans":[{"name":"identity"}],"threadId":"ThreadId(2)"}

output of linkerd check -o short

Linkerd core checks
===================

linkerd-existence
-----------------
/ No running pods for "linkerd-destination"

Environment

• Kubernetes Version: 1.21
• Cluster Environment: AWS EKS
• Host OS: linux (amd64)
• Linkerd Version: 2.11.1

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

No response

[olix0r]

It sounds like the identity controller is misconfigured. Is there anything in its logs? How are the issuer certificates configured via cert-manager?

[dreinhardt89]

@olix0r Have some more info.

Configuration (helm/terraform):

locals {
  linkerd_ca_helm_release_values = {
    "certificate.commonName" = "identity.linkerd.cluster.local",
    "certificate.secretName" = "linkerd-identity-issuer",
    trustAnchorSecret        = kubernetes_secret.linkerd_trust_anchor[0].metadata[0].name,
    } 
  }

resource "helm_release" "linkerd_cert" {
  name            = "linkerd-cert"
  chart           = "${path.module}/helm/charts/ca-cert"
  namespace       = var.namespace
  atomic          = true
  cleanup_on_fail = true
  force_update    = true

  dynamic "set" {
    for_each = local.linkerd_ca_helm_release_values

    content {
      name  = set.key
      value = set.value
    }
  }
}

values.yaml:

enabled: true

certificate:
  secretName: ~
  duration: 2160h # 90 days
  renewBefore: 336h
  commonName: ~
  isCA: true
  usages:
    - cert sign
    - crl sign
    - server auth
    - client auth

trustAnchorSecret: ~

certificate.yaml:

{{- if .Values.enabled -}}
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-identity-issuer
  labels:
    {{- include "ca-cert.labels" . | nindent 4 }}
spec:
  secretName: {{ include "ca-cert.secret" . }}
  duration: {{ .Values.certificate.duration }}
  renewBefore: {{ .Values.certificate.renewBefore }}
  issuerRef:
    name: {{ include "ca-cert.trust-anchor" . }}
    kind: Issuer
  commonName: {{ .Values.certificate.commonName }}
  isCA: {{ .Values.certificate.isCA }}
  keyAlgorithm: {{ .Values.certificate.keyAlgorithm }}
  usages:
    {{- toYaml .Values.certificate.usages | nindent 4}}
{{- end -}}

issuer.yaml:

{{- if .Values.enabled -}}
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: {{ include "ca-cert.trust-anchor" . }}
  labels:
    {{- include "ca-cert.labels" . | nindent 4 }}
spec:
  ca:
    secretName: {{ .Values.trustAnchorSecret }}
{{- end -}}

_helpers.tpl:

{{/* vim: set filetype=mustache: */}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "ca-cert.fullname" -}}
{{- printf "%s" .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "ca-cert.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{/*
Common labels
*/}}
{{- define "ca-cert.labels" -}}
helm.sh/chart: {{ include "ca-cert.chart" . }}
app.kubernetes.io/name: {{ .Release.Name }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}

{{/*
Certificate secret name
*/}}
{{- define "ca-cert.secret" -}}
{{- $name := printf "%s-identity-issuer" (include "ca-cert.fullname" .) -}}
{{- printf "%s" (default $name .Values.certificate.secretName) -}}
{{- end -}}

{{/*
Trust anchor name
*/}}
{{- define "ca-cert.trust-anchor" -}}
{{ include "ca-cert.fullname" . }}-trust-anchor
{{- end -}}

{{/*
Identity issuer name
*/}}
{{- define "ca-cert.identity-issuer" -}}
{{ include "ca-cert.fullname" . }}-identity-issuer
{{- end -}}

from kubectl get certificate -n linkerd:

linkerd-cert-identity-issuer   True    linkerd-identity-issuer   11h

from kubectl -n linkerd get issuer -o yaml:

apiVersion: v1
items:
- apiVersion: cert-manager.io/v1
  kind: Issuer
  metadata:
    annotations:
      meta.helm.sh/release-name: linkerd-cert
      meta.helm.sh/release-namespace: linkerd
    creationTimestamp: "2022-08-04T06:18:46Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: linkerd-cert
      app.kubernetes.io/version: v0.14.3
      helm.sh/chart: ca-cert-0.1.6
    name: linkerd-cert-trust-anchor
    namespace: linkerd
    resourceVersion: "150092"
    uid: c77320cd-a1e7-41bc-a554-e3a41b6dd7a2
  spec:
    ca:
      secretName: linkerd-trust-anchor
  status:
    conditions:
    - lastTransitionTime: "2022-08-04T06:18:46Z"
      message: Signing CA verified
      observedGeneration: 1
      reason: KeyPairVerified
      status: "True"
      type: Ready
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""