skaffold dev does not rebuild image when using kind with registry

[annadcunningham]

Expected behavior

When running skaffold dev using a kind cluster with local registry enabled, editing the Dockerfile should cause the image to rebuild and the container should be re-deployed with the updated image.

Actual behavior

The image is not rebuilt.

Information

• Skaffold version: v2.10.0
• Operating system: MacOS Sonoma 14.3
• Installed via: Homebrew
• Contents of skaffold.yaml: Using the getting started example

Steps to reproduce the behavior

1. In this repo, cd examples/getting-started
2. Create a kind cluster with registry using this script: https://kind.sigs.k8s.io/docs/user/local-registry/
3. skaffold dev -v info
4. Edit the Dockerfile, for example add this line to the bottom: RUN touch test.txt
5. View the logs; the docker image does not rebuild.
6. To confirm the test.txt was not created in the container: kubectl exec -it getting-started -- ls
7. To clean up the kind cluster and registry, run kind delete clusters "kind"; docker network disconnect --force kind kind-registry; docker container rm --force --volumes kind-registry

Logs from skaffold dev -v info:

INFO[0001] Skaffold &{Version:v2.10.0 ConfigVersion:skaffold/v4beta9 GitVersion: GitCommit:cbc665bfc1fe7253df466e70dd48e3851d935a3e BuildDate:2024-01-09T08:59:34Z GoVersion:go1.21.0 Compiler:gc Platform:darwin/arm64 User:}  subtask=-1 task=DevLoop
INFO[0001] Loaded Skaffold defaults from "/Users/anna.cunningham/.skaffold/config"  subtask=-1 task=DevLoop
INFO[0001] map entry found when executing locate for &{skaffold-example . <nil> {0x140009fe160 <nil> <nil> <nil> <nil> <nil> <nil>} [] {[] []} [] } of type *latest.Artifact and pointer: 1374399343456  subtask=-1 task=DevLoop
INFO[0001] Using kubectl context: kind-kind              subtask=-1 task=DevLoop
INFO[0001] using default-repo=localhost:5001 from cluster configmap  subtask=-1 task=DevLoop
INFO[0001] DOCKER_HOST env is not set, using the host from docker context.  subtask=-1 task=DevLoop
INFO[0001] no kpt renderer or deployer found, skipping hydrated-dir creation  subtask=-1 task=DevLoop
INFO[0001] build concurrency first set to 1 parsed from *runner.pipelineBuilderWithHooks[0]  subtask=-1 task=DevLoop
INFO[0001] final build concurrency value is 1            subtask=-1 task=DevLoop
Generating tags...
 - skaffold-example -> localhost:5001/skaffold-example:v2.10.0-23-geeaf00144
INFO[0001] Tags generated in 93.908666ms                 subtask=-1 task=Build
Checking cache...
 - skaffold-example: Not found. Building
INFO[0003] Cache check completed in 1.985 second         subtask=-1 task=Build
Starting build...
Found [kind-kind] context, using local docker daemon.
Building [skaffold-example]...
Target platforms: [linux/arm64]
#0 building with "orbstack" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 510B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/golang:1.18
#2 DONE 0.8s

#3 [internal] load metadata for docker.io/library/alpine:3
#3 DONE 0.8s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [builder 1/5] FROM docker.io/library/golang:1.18@sha256:50c889275d26f816b5314fc99f55425fa76b18fcaf16af255f5d57f09e1f48da
#5 DONE 0.0s

#6 [stage-1 1/2] FROM docker.io/library/alpine:3@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b
#6 DONE 0.0s

#7 [internal] load build context
#7 transferring context: 54B done
#7 DONE 0.0s

#8 [builder 2/5] WORKDIR /code
#8 CACHED

#9 [builder 4/5] COPY go.mod .
#9 CACHED

#10 [builder 5/5] RUN go build -gcflags="${SKAFFOLD_GO_GCFLAGS}" -trimpath -o /app main.go
#10 CACHED

#11 [builder 3/5] COPY main.go .
#11 CACHED

#12 [stage-1 2/2] COPY --from=builder /app .
#12 CACHED

#13 exporting to image
#13 exporting layers done
#13 writing image sha256:eb4f7b79eb739540a81d000ed323c739b22d90435d8b314e833db1c8c7876ef7 done
#13 naming to localhost:5001/skaffold-example:v2.10.0-23-geeaf00144 done
#13 DONE 0.0s
The push refers to repository [localhost:5001/skaffold-example]
445ce1b3ac91: Preparing
b09314aec293: Preparing
445ce1b3ac91: Layer already exists
b09314aec293: Layer already exists
v2.10.0-23-geeaf00144: digest: sha256:1a77ed36ef1f81282f51e6f8162231d7cea094a74ee0bd98633c1008cf2c8def size: 738
Build [skaffold-example] succeeded
INFO[0005] Build completed in 1.575 second               subtask=-1 task=Build
INFO[0005] Starting render...                            subtask=-1 task=DevLoop
INFO[0005] starting render process                       subtask=0 task=Render
INFO[0005] Render completed in 2.1945ms                  subtask=-1 task=DevLoop
Tags used in deployment:
 - skaffold-example -> localhost:5001/skaffold-example:v2.10.0-23-geeaf00144@sha256:1a77ed36ef1f81282f51e6f8162231d7cea094a74ee0bd98633c1008cf2c8def
Starting deploy...
 - pod/getting-started created
Waiting for deployments to stabilize...
 - pods is ready.
Deployments stabilized in 3.102 seconds
INFO[0008] Deploy completed in 3.318 seconds             subtask=-1 task=Deploy
Listing files to watch...
 - skaffold-example
INFO[0008] List generated in 101.125µs                   subtask=-1 task=DevLoop
Press Ctrl+C to exit
Watching for changes...
INFO[0008] Streaming logs from pod: getting-started container: getting-started  subtask=-1 task=DevLoop
[getting-started] Hello world!
[getting-started] Hello world!

then after editing the Dockerfile:

INFO[0022] files modified: [Dockerfile]                  subtask=-1 task=DevLoop
Generating tags...
 - skaffold-example -> localhost:5001/skaffold-example:v2.10.0-23-geeaf00144-dirty
INFO[0022] Tags generated in 88.819792ms                 subtask=-1 task=Build
Checking cache...
 - skaffold-example: Found Remotely
INFO[0022] Cache check completed in 29.823333ms          subtask=-1 task=Build
INFO[0022] Starting render...                            subtask=-1 task=DevLoop
INFO[0022] starting render process                       subtask=0 task=Render
INFO[0022] Render completed in 3.210583ms                subtask=-1 task=DevLoop
Tags used in deployment:
 - skaffold-example -> localhost:5001/skaffold-example:v2.10.0-23-geeaf00144-dirty@sha256:8e32f5ea7a6f02e102714554c1b7fc73f8b0968452bdb89744ac7c9a2610fd59
Starting deploy...
 - pod/getting-started configured
Waiting for deployments to stabilize...
INFO[0023] Streaming logs from pod: getting-started container: getting-started  subtask=-1 task=DevLoop
 - pods is ready.
Deployments stabilized in 2.11 seconds
INFO[0024] Deploy completed in 2.323 seconds             subtask=-1 task=Deploy
Watching for changes...
[getting-started] Hello world!

On version 2.9.0, editing the Dockerfile results in these logs; I would expect 2.10.* to have the same behavior:

INFO[0015] files modified: [Dockerfile]                  subtask=-1 task=DevLoop
Generating tags...
 - skaffold-example -> localhost:5001/skaffold-example:v2.10.0-23-geeaf00144-dirty
INFO[0015] Tags generated in 85.997584ms                 subtask=-1 task=Build
Checking cache...
 - skaffold-example: Not found. Building
INFO[0015] Cache check completed in 1.612458ms           subtask=-1 task=Build
Starting build...
Found [kind-kind] context, using local docker daemon.
Building [skaffold-example]...
Target platforms: [linux/arm64]
#0 building with "orbstack" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 530B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/alpine:3
#2 DONE 0.4s

#3 [internal] load metadata for docker.io/library/golang:1.18
#3 DONE 0.4s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [builder 1/5] FROM docker.io/library/golang:1.18@sha256:50c889275d26f816b5314fc99f55425fa76b18fcaf16af255f5d57f09e1f48da
#5 DONE 0.0s

#6 [stage-1 1/3] FROM docker.io/library/alpine:3@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b
#6 DONE 0.0s

#7 [internal] load build context
#7 transferring context: 54B done
#7 DONE 0.0s

#8 [builder 2/5] WORKDIR /code
#8 CACHED

#9 [builder 4/5] COPY go.mod .
#9 CACHED

#10 [builder 3/5] COPY main.go .
#10 CACHED

#11 [builder 5/5] RUN go build -gcflags="${SKAFFOLD_GO_GCFLAGS}" -trimpath -o /app main.go
#11 CACHED

#12 [stage-1 2/3] COPY --from=builder /app .
#12 CACHED

#13 [stage-1 3/3] RUN touch test4.txt
#13 DONE 0.1s

#14 exporting to image
#14 exporting layers done
#14 writing image sha256:5efd379ca49ef09490648c2edbdbc3a556687492a3987bd586e39d5092a05e29 done
#14 naming to localhost:5001/skaffold-example:v2.10.0-23-geeaf00144-dirty done
#14 DONE 0.0s
The push refers to repository [localhost:5001/skaffold-example]
69f140f181ce: Preparing
445ce1b3ac91: Preparing
b09314aec293: Preparing
445ce1b3ac91: Layer already exists
b09314aec293: Layer already exists
69f140f181ce: Pushed
v2.10.0-23-geeaf00144-dirty: digest: sha256:39697e8f6a3cf265ccc23fd0b230b97c2cbfcd7011566237707dfcac3a6f5bc9 size: 945
Build [skaffold-example] succeeded
INFO[0016] Build completed in 973.0615ms                 subtask=-1 task=Build
INFO[0016] Starting render...                            subtask=-1 task=DevLoop
INFO[0016] starting render process                       subtask=0 task=Render
INFO[0016] Render completed in 1.511ms                   subtask=-1 task=DevLoop
Tags used in deployment:
 - skaffold-example -> localhost:5001/skaffold-example:v2.10.0-23-geeaf00144-dirty@sha256:39697e8f6a3cf265ccc23fd0b230b97c2cbfcd7011566237707dfcac3a6f5bc9
Starting deploy...
 - pod/getting-started configured
Waiting for deployments to stabilize...
INFO[0017] Streaming logs from pod: getting-started container: getting-started  subtask=-1 task=DevLoop
 - pods is ready.
Deployments stabilized in 2.111 seconds
INFO[0018] Deploy completed in 2.267 seconds             subtask=-1 task=Deploy
Watching for changes...
[getting-started] Hello world!
[getting-started] Hello world!

[pre]

We have noticed the same behaviour and can confirm the issue as well.

[rajatvig]

Same. We have noticed the same issue in 2.10

[bite-bruce]

Same issue here, I have to foce delete the whole docker hub repository to valid local image cache, but sometimes it doesn't works as well

[B11PP]

We have the same issue when using k0s with a registry.

[Mwoab]

Same trouble here. We use skaffold Version:v2.10.1 embedded in the Goland Google Cloud Code plugin. We are force to git push (to generate a new tag) or to remove manually the related dirty image of the repitory (Google Artifact Registry).

[Mwoab]

Seems to be fixed with the v2.11.0 which is now used with the Goland Google Cloud Code plugin