How to load locally built images into Kind?

[iiian]

I'm in a similar situation to this ticket: >>How to use locally built docker image?<<, but I'm coming from Kind instead. Additionally:

1. I tried applying most of the advice in that ticket, but was still facing the same issues in minikube, and so I decided to switch.
2. I found @xphoniex's fork with instructions to run local docker images, but I am a bit concerned that it's not something captured anymore in the knative documentation website (tried to trace down the PR which erased it, looks like it was there at some point, but I wasn't dedicated enough to complete the detective work).

Either way, seems like both Kind and Minikube should be documented for this? Anyway, in Kind's documentation, they suggest using the following script to set up an external docker registry and then link it via plugin & docker network: https://kind.sigs.k8s.io/docs/user/local-registry/

The plugin is plugins."io.containerd.grpc.v1.cri".registry.mirrors.

I have a script which combines ☝️ with Install Serving With YAML to build a Kind cluster with a local docker registry@ localhost:5001, + kn goodness:

File: start-kn-kub.sh

#-------------------------------------------------------------------------
# FROM: https://kind.sigs.k8s.io/docs/user/local-registry/

set -o errexit

# create registry container unless it already exists
reg_name='kind-registry'
reg_port='5001'
if [ "$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)" != 'true' ]; then
  docker run \
    -d --restart=always -p "127.0.0.1:${reg_port}:5000" --name "${reg_name}" \
    registry:2
fi

# create a cluster with the local registry enabled in containerd
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:${reg_port}"]
    endpoint = ["http://${reg_name}:5000"]
EOF

# connect the registry to the cluster network if not already connected
if [ "$(docker inspect -f='{{json .NetworkSettings.Networks.kind}}' "${reg_name}")" = 'null' ]; then
  docker network connect "kind" "${reg_name}"
fi

# Document the local registry
# https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-registry-hosting
  namespace: kube-public
data:
  localRegistryHosting.v1: |
    host: "localhost:${reg_port}"
    help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
EOF

#-------------------------------------------------------------------------
# FROM: https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#verify-the-installation
echo 🔥 Installing Knative Eventing v1.3.0 ...
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.3.0/serving-crds.yaml
echo CRDs installed...
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.3.0/serving-core.yaml
echo Core installed...
kubectl apply -f https://github.com/knative/net-kourier/releases/download/knative-v1.3.0/kourier.yaml

echo 🕸 Configuring Kourier for Kind...
kubectl apply -f https://github.com/knative/net-kourier/releases/download/knative-v1.3.0/kourier.yaml
kubectl patch configmap/config-network \
  --namespace knative-serving \
  --type merge \
  --patch '{"data":{"ingress.class":"kourier.ingress.networking.knative.dev"}}'
kubectl --namespace kourier-system get service kourier
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.3.0/serving-default-domain.yaml

I am able to create a deployment which references an image localhost:5001/hello-world in the registry. This image is https://hub.docker.com/r/crccheck/hello-world/, and is created by docker tag localhost:5001/hello-world crccheck/hello-world. I am confident that the image exists, as I can observe both the registry servicing a ingesting it in docker desktop, as well as ktl get pods reporting its existence & that it is running after creating a deployment from it.

Things break down when I try to run any of the following:

kn service create hello-world --image localhost:5001/hello-world
kn service create hello-world --image dev.local/hello-world
kn service create hello-world --image ko.local/hello-world
kn service create hello-world --image hello-world

the first results in:

kn service create s0
 --image localhost:5001/hello-world --revision-name=v1
Creating service 's0' in namespace 'default':

  0.005s The Configuration is still working to reflect the latest desired specification.
  0.024s The Route is still working to reflect the latest desired specification.
  0.048s Revision "s0-v1" failed with message: Unable to fetch image "localhost:5001/hello-world": failed to resolve image to digest: Get "https://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused; Get "http://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused.
  0.052s ...
  0.069s Configuration "s0" does not have any ready Revision.
Error: RevisionFailed: Revision "s0-v1" failed with message: Unable to fetch image "localhost:5001/hello-world": failed to resolve image to digest: Get "https://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused; Get "http://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused.
Run 'kn --help' for usage

the rest are similar.

Additionally, I have tried both kind load docker-image localhost:5001/hello-world and docker push localhost:5001/hello-world, but neither seem to make a difference. The image is definitely there though.

I was assuming that, given a kubernetes environment with an overridden docker registry loaded with my image, I'd be able to just invoke it. Additionally, I saw >>this comment<< (copied below) on the main ticket:

... A docker image without a hostname prefix is assumed to be DockerHub. Your image pull policy of 'Never' prevents minikube from reaching out to DockerHub to try to fetch the image and assumes that the version you have locally is the latest and greatest.

The last step should have been successful with an Image Pull policy of 'Never'. What I suspect might have went wrong here is that you edited the existing line. The existing line is part of an _example key and modifying it wont take effect. You need to copy the line out of the example and move it up to be its own key. ...

I didn't understand the "existing _example line" thing, but am wondering if that could be a part of my issue? Maybe that was part of a side conversation.

Anything helps, as I'm still in my first few weeks of Kubernetes, transitioning from AWS lambda+CDK. I won't totally need to be able to run kubernetes and knative for my local development set up, but it would certainly be nice if I we ever decide we need to do some e2e tests.

Thanks!

[psschwei]

I didn't understand the "existing _example line" thing, but am wondering if that could be a part of my issue? Maybe that was part of a side conversation.

What this means... the configmap looks something like this:

data:
  # This is the Go import path for the binary that is containerized
  # and substituted here.
  # TODO: switch to 'queue-sidecar-image' after 0.27
  queueSidecarImage: ko://knative.dev/serving/cmd/queue
  _example: |-
    ################################
    #                              #
    #    EXAMPLE CONFIGURATION     #
    #                              #
    ################################
    # This block is not actually functional configuration,
    # but serves to illustrate the available configuration
    # options and document them in a way that is accessible
    # to users that `kubectl edit` this config map.
    #
    # These sample configuration options may be copied out of
    # this example block and unindented to be in the data block
    # to actually change the configuration.
    # List of repositories for which tag to digest resolving should be skipped
    registries-skipping-tag-resolving: "kind.local,ko.local,dev.local"

You can't just add your local registry to the existing list, i.e. something like this:

data:
  # This is the Go import path for the binary that is containerized
  # and substituted here.
  # TODO: switch to 'queue-sidecar-image' after 0.27
  queueSidecarImage: ko://knative.dev/serving/cmd/queue
  _example: |-
    ################################
    #                              #
    #    EXAMPLE CONFIGURATION     #
    #                              #
    ################################
    # This block is not actually functional configuration,
    # but serves to illustrate the available configuration
    # options and document them in a way that is accessible
    # to users that `kubectl edit` this config map.
    #
    # These sample configuration options may be copied out of
    # this example block and unindented to be in the data block
    # to actually change the configuration.
    # List of repositories for which tag to digest resolving should be skipped
    registries-skipping-tag-resolving: "kind.local,ko.local,dev.local,localhost:5001"

(note the update to the last line)

Instead, you need to add it outside of the _example block, i.e.

data:
  # This is the Go import path for the binary that is containerized
  # and substituted here.
  # TODO: switch to 'queue-sidecar-image' after 0.27
  queueSidecarImage: ko://knative.dev/serving/cmd/queue
  registries-skipping-tag-resolving: "localhost:5001"
  _example: |-
    ################################
    #                              #
    #    EXAMPLE CONFIGURATION     #
    #                              #
    ################################
    # This block is not actually functional configuration,
    # but serves to illustrate the available configuration
    # options and document them in a way that is accessible
    # to users that `kubectl edit` this config map.
    #
    # These sample configuration options may be copied out of
    # this example block and unindented to be in the data block
    # to actually change the configuration.
    # List of repositories for which tag to digest resolving should be skipped
    registries-skipping-tag-resolving: "kind.local,ko.local,dev.local"

The 6th line (the one beginning registries-skipping-tag-resolving...), that's the one you'll need to add.

With that addition to the configmap, I was able to deploy a Knative service (using image localhost:5001/helloworld-go from the local registry and imagePullPolicy: Never) to a local kind cluster.

Hope that helps!

[dgeorgievski]

I was about to submit an issue with the project to inform others of how to successfully configure a local KinD cluster with Knative and Kourier, when I saw this issue :-)

I'd advise against using local registry with KinD. It just creates new challenges depending on your platform For example Mac, that I am using along with Ubuntu, imposes networking restrictions that would make it hard if not impossible to access the local registry simultaneously from the laptop and KinD cluster, also running in another docker container. It is possible but unnecessary because it will complicate significantly the setup of your local Dev environment.

The much simpler solution is to disable tag resolution for your images loaded into the KinD nodes. kubectl -n knative-serving edit cm config-deployment

and make sure registries-skipping-tag-resolving setting is enabled

apiVersion: v1
kind: ConfigMap
name: config-deployment
namespace: knative-serving
data:
  registries-skipping-tag-resolving: kind.local,ko.local,dev.local,foo.com,index.docker.io
  _example: |-
  ...

The trick was to add index.docker.io as a reference to Docker Hub registry. I haven't tried yet the other registries nameskind.local,ko.local,dev.local,foo.com

After this change I was able to deploy Releases successfully in my local KinD cluster

$ kn service apply -f manifests/service.yaml
Creating service 'taina' in namespace 'default':

  0.234s The Route is still working to reflect the latest desired specification.
  0.343s ...
  0.514s Configuration "taina" is waiting for a Revision to become ready.
  4.604s ...
  4.848s Ingress has not yet been reconciled.
  5.129s Waiting for load balancer to be ready
  5.213s Ready to serve.

Service 'taina' created to latest revision 'taina-00001' is available at URL:
http://taina.default.127.0.0.1.sslip.io/

$ kn service list
NAME            URL                                               LATEST                AGE   CONDITIONS   READY   REASON
taina           http://taina.default.127.0.0.1.sslip.io/           taina-00001           19s   3 OK / 3     True

You can see the full details in this doc https://github.com/wiley/taina/blob/main/kind/README.md

[iiian]

@psschwei thanks; I'll give that a shot.

@dgeorgievski if I do that, won't I have to specify digests instead of image names when creating kservices?

[dgeorgievski]

if I do that, won't I have to specify digests instead of image names when creating kservices?

No, it is not necessary. Knatiive Serving does that - it resolves a tag to a digest by sending a HEAD request to the remote registry .. unless configured not to do so.

Check this for more details https://docs.google.com/presentation/d/e/2PACX-1vTgyp2lGDsLr_bohx3Ym_2mrTcMoFfzzd6jocUXdmWQFdXydltnraDMoLxvEe6WY9pNPpUUvM-geJ-g/pub?resourcekey=0-FH5lN4C2sbURc_ds8XRHeA&slide=id.g50ae5edc09_0_380

And this from Docker Registry doc https://docs.docker.com/registry/spec/api/#content-digests

A copy of knative revision messages from failed attempts to deploy my service into KinD illustrating this point.

0.447s Configuration "taina" is waiting for a Revision to become ready.
 11.353s Revision "taina-00001" failed with message: Unable to fetch image "docker.io/fatcatsart/taina:3ede274": failed to resolve image to digest: Head "https://index.docker.io/v2/fatcatsart/taina/manifests/3ede274": context deadline exceeded.

[iiian]

I was able to apply your advice @psschwei!

kn service create hello --image localhost:5001/hello-world --revision-name=v1 --port 8080 
Creating service 'hello' in namespace 'default':

  0.007s The Configuration is still working to reflect the latest desired specification.
  0.024s The Route is still working to reflect the latest desired specification.
  0.031s Configuration "hello" is waiting for a Revision to become ready.
  1.807s ...
  1.825s Ingress has not yet been reconciled.
  1.850s Waiting for load balancer to be ready
  2.040s Ready to serve.

Service 'hello' created to latest revision 'hello-v1' is available at URL:
http://hello.default.example.com

However, now when I try to connect to http://hello.default.example.com (yes it does give me the .example and not sslip.io, maybe that's an issue), browser, curl, postman all fail to resolve this host. Hmm.

[psschwei]

However, now when I try to connect to http://hello.default.example.com (yes it does give me the .example and not sslip.io, maybe that's an issue), browser, curl, postman all fail to resolve this host. Hmm.

As a workaround, you can use the "Temporary DNS" option if the magic DNS isn't working.

[iiian]

@dgeorgievski okay so I worked through the doc tutorial you shared. I am a bit confused--I didn't understand if you were using a local image or not. I set up the registries-skipping-tag-resolving: kind.local,ko.local,dev.local,foo.com,index.docker.io as prescribed. Then I ran

$ kind load docker-image hello-world;
Image: "hello-world" with ID "sha256:b86234f4a1ae2bfd4301e95f0805b4d10442b05cd904700fc56e04befb90b46a" not yet present on node "knative-control-plane", loading...

$ kn service create hello --image hello-world
Creating service 'hello' in namespace 'default':

  0.010s The Configuration is still working to reflect the latest desired specification.
  0.025s The Route is still working to reflect the latest desired specification.
  0.047s Configuration "hello" is waiting for a Revision to become ready.

However, the process is hung here, waiting for a revision to become ready.

Also, didn't totally understand the significance of the taina thing (oh I guess that's your project).

[iiian]

I've now tried to replicate the original ticket's approach to serving in its entirety, but cannot achieve success:

# 1.
kn quickstart minikube
# 2.
kubectl edit configmaps config-deployment --namespace knative-serving 
#     -- at this step, I ensure that `dev.local,minikube,mini.kube` are all 
#        listed in the `registries-skipping-tag-resolving` property.
#
# 3.
eval $(minikube docker-env)
# 4.
docker build -t dev.local/my-service:v1 .

#And then finally, I have tried both of the following forms of kn service deployment:

# 5a.
kubectl apply -f service.yaml
# where service.yaml:
# apiVersion: serving.knative.dev/v1 # Current version of Knative
# kind: Service
# metadata:
#   name: my-service # The name of the app
#   namespace: default # The namespace the app will use
# spec:
#   template:
#     spec:
#       containers:
#         - image: dev.local/my-service:v1
#           imagePullPolicy: Never

# 5b.
kn service create my-service --image=dev.local/my-service:v1

# but both 5a & 5b get hung "waiting for revision to show up".

Any thoughts?

[xtaje]

Here's a step-by-step tutorial for kind based on these comments https://github.com/knative/docs/pull/4948

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.