Failed to use local registry with Ratify on K3s and Minukube

[FeynmanZhou]

I tried to use Notation and the local registry (CNCF distribution/distribution) with Ratify to verify a signed image on K3s and Minukube, but the verification failed. The error logs from the Ratify pod are as follows:

On Minikube:

time="2023-06-06T08:55:05Z" level=info msg="Resolve of the image completed successfully the digest is sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b"
{
  "isSuccess": false,
  "verifierReports": [
    {
      "isSuccess": false,
      "name": "notaryv2",
      "message": "an error thrown by the verifier: failed to verify signature, err: signature is not produced by a trusted signer",
      "artifactType": "application/vnd.cncf.notary.signature"
    }
  ]
}
time="2023-06-06T08:55:16Z" level=info msg="received request POST /ratify/gatekeeper/v1/mutate "
time="2023-06-06T08:55:16Z" level=info msg="start request POST /ratify/gatekeeper/v1/mutate"
time="2023-06-06T08:55:16Z" level=info msg="mutating image localhost:5001/net-monitor@sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e"
time="2023-06-06T08:55:16Z" level=info msg="received request POST /ratify/gatekeeper/v1/verify "
time="2023-06-06T08:55:16Z" level=info msg="start request POST /ratify/gatekeeper/v1/verify"
time="2023-06-06T08:55:16Z" level=info msg="verifying subject localhost:5001/net-monitor@sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e"
time="2023-06-06T08:55:16Z" level=warning msg="failed to resolve the subject descriptor from store oras with error Head \"https://localhost:5001/v2/net-monitor/manifests/sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e\": dial tcp 127.0.0.1:5001: connect: connection refused\n"
{
  "isSuccess": false,
  "verifierReports": [
    {
      "subject": "localhost:5001/net-monitor@sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e",
      "isSuccess": false,
      "message": "verification failed: resolving descriptor for the subject failed with error: could not resolve descriptor for a subject from any stores"
    }
  ]
}

On K3s

time="2023-06-08T10:40:57Z" level=warning msg="failed to resolve the subject descriptor from store oras with error Head \"[https://localhost:5001/v2/net-monitor/manifests/sha256:1c6c22b925c7f1e7d41f1fdb3a8723c3a01c5299f20c07080c6f9db96018cb11\":](https://localhost:5001/v2/net-monitor/manifests/sha256:1c6c22b925c7f1e7d41f1fdb3a8723c3a01c5299f20c07080c6f9db96018cb11/%22:) dial tcp [::1]:5001: connect: connection refused\n"
{
  "isSuccess": false,
  "verifierReports": [
    {
      "subject": "localhost:5001/net-monitor@sha256:1c6c22b925c7f1e7d41f1fdb3a8723c3a01c5299f20c07080c6f9db96018cb11",
      "isSuccess": false,
      "message": "verification failed: resolving descriptor for the subject failed with error: could not resolve descriptor for a subject from any stores"
    }
  ]
}

It seems like a network issue with connecting the local registry. Is there any additional configuration I missed?

Another question is that I find it is hard to troubleshoot Ratify by inspecting the logs since the logs may come from notation-go, oras-go, or Ratify. There is an issue tracking a similar problem in #856 . How should users distinguish the source of the logs and where can users find the troubleshooting guideline? I think we need to consolidate the logs from different sources and provide some common wrapped error messages for debugging conveniently.

[akashsinghal]

This looks like an issue with setup of the local registry. How is the registry deployed? Is it running as a pod in the cluster or is it running as a standalone container on your computer?

[binbin-li]

This looks like an issue with setup of the local registry. How is the registry deployed? Is it running as a pod in the cluster or is it running as a standalone container on your computer?

I also tried on Minikube with registry deployed outside of the cluster, which got the same error. I followed some doc that enabled insecure-registry option while starting minikube which didn't work. I feel it might be something missing with Minikube configuration.

[FeynmanZhou]

@akashsinghal @binbin-li My local registry is deployed as a standalone container on my computer.

docker run -d -p 5001:5000 -e REGISTRY_STORAGE_DELETE_ENABLED=true --name registry registry

[akashsinghal]

@FeynmanZhou it looks like there isn't an easy way to do this unfortunately. There's 2 ways to do this:

1. The registry is running on your local computer and you need to setup the correct routing so the minikube cluster can resolve the localhost registry. (this is the setup you currently are trying to achieve) Here'sa walkthrough to get it working
2. Deploy the registry inside minikube and then port forward the registry service endpoint to the host computer. Here's a walkthrough.

[yizha1]

@FeynmanZhou is this still blocking you?

[FeynmanZhou]

Yes, we still need to validate it and create a dedicated document to guide users on how to use Ratify with a local registry on a local K8s distro. This is not a priority for now so we may defer it to v1.0.0.