Error getting root certificate with LinkedCA deployment on latest version

[xinbinhuang]

Subject of the issue

I'm using Linked CA deployment. Recently, upgrade the helm chart version to the latest but since then getting this error on start and crashlooped.

badger 2023/09/25 18:32:45 INFO: All 1 tables opened in 10ms                                                                                                                                                                                                                                                                                
badger 2023/09/25 18:32:45 INFO: Replaying file id: 0 at offset: 109765353                                                                                                                                                                                                                                                                  
badger 2023/09/25 18:32:46 INFO: Replay took: 191.535059ms                                                                                                                                                                                                                                                                                  
error getting root certificate: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"

Your environment

• OS - Ubuntu
• Version - 1.24.2+1

Steps to reproduce

Tell us how to reproduce this issue. Please provide a working demo, you can use this template as a base.

Upgrade from 1.23.2+5 to 1.24.2+1

Expected behaviour

Tell us what should happen.

No error

Actual behaviour

Tell us what happens instead.

Additional context

Add any other context about the problem here.

[maraino]

Hi @xinbinhuang, I'm looking at the logs and I don't see any request returning that error, I wonder if it was a temporary problem with an ingress.

Is this something you can reproduce it always?

Can you try to use grpcurl from the same environment to make sure it's not a connection issue at your end?

The command would be:

$ grpcurl -insecure -d '{"fingerprint": "2abb04350794befa232a6229bcddbd945e9408a20490f699a6538343c3a6b608"}' linkedca.api.smallstep.com:443 linkedca.Majordomo/GetRootCertificate

You will get the root certificate used by step-ca and the linkedca server to build the root of trust.

If you inspect your token cat token.txt | step crypto jwt inspect --insecure you will see that fingerprint as the sha attribute.

[xinbinhuang]

Hey @maraino , thanks for the response. I tried to run the command and it did return the root certificate without issue.

kubectl run test-step --image fullstorydev/grpcurl:latest -- -insecure -d '{"fingerprint": "<fingerprint>"}' linkedca.api.smallstep.com:443 linkedca.Majordomo/GetRootCertificate

// log response
{"pemCertificate" : "..."}

[xinbinhuang]

these are the values for my deployment

---
# Default values for step-certificates.

# kind is the type of object to use when deploying the CA.
# Changing the deployment type is experimental.
kind: StatefulSet

# replicaCount is the number of replicas of step-certificates.
# Only one replica is supported at this time.
replicaCount: 1

# nameOverride overrides the name of the chart.
nameOverride: ''
# fullnameOverride overrides the full name of the chart.
fullnameOverride: ''

# image contains the docker image for step-certificates.
image:
  repository: cr.step.sm/smallstep/step-ca
  initContainerRepository: busybox:latest
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: ""

# Use existing secrets instead of 'bootstrap' init-container or 'inject'-element from helm chart
# Note, the MutatingWebhookConfiguration created by autocert is not patched with CA bundle as the bootstrap init-container is not run
existingSecrets:
  enabled: true
  ca: true
  configAsSecret: true

bootstrap:
  secrets: false
  enabled: false
  configmaps: false

inject:
  enabled: false

# linkedca contains the token to configure step-ca using the linkedca mode.
#
# The linked ca token can be provided using the linkedca.token value or using a
# reference to a secret.
linkedca:
  # Reference to a secret name and key.
  secretKeyRef:
    name: step-certificates-linkedca-token
    key: linkedca-token

# ca contains the certificate authority configuration.
ca:
  # db contains the step-certificate database configuration.
  db:
    # enabled defines if the database is enabled.
    enabled: true
    # persistent defines if a Persistent Volume Claim is used, if false and emptyDir will be used.
    persistent: true
    # storeageClass is Persistent Volume Storage Class
    # If defined, storageClassName: <storageClass>.
    # If set to "-", storageClassName: "", which disables dynamic provisioning.
    # If undefined or set to null, no storageClassName spec is set, choosing the
    # default provisioner (gp2 on AWS, standard on GKE, AWS & OpenStack).
    storageClass: longhorn
    ## Persistent Volume existing claim name
    ## Requires ca.db.persistent: true
    ## If defined, PVC must be created manually before volume will be bound
    # existingClaim: ""
    # accessModes defines the Persistent Volume Access Mode.
    accessModes:
      - ReadWriteOnce
    # size is the Persistent Volume size.
    size: 10Gi

# ingress contains the configuration for an ingress controller.
ingress:
  enabled: false
  annotations: {}
  ingressClassName: ''
  hosts: []
  tls: []