tilt icon indicating copy to clipboard operation
tilt copied to clipboard

Tilt fails to pull Docker images with k8s_yaml - It tries to pull images (with a `:tilt-<hash>` tag) from docker.io

Open Hiroki111 opened this issue 3 months ago • 5 comments

Expected Behavior

I expect Tilt to

  1. build Docker images with docker_build,
  2. register a Helm chart by k8s_yaml(helm('directory/to/chart')) (or k8s_yaml(local('helm template directory/to/chart'))),
  3. and then deploy them in K8s cluster running in Docker Desktop

Current Behavior

I can see that Docker images are built and a Helm chart is registered. However, Tilt can't pull the images. Tilt seems to be trying to pull an image with a :tilt-<hash> tag from docker.io, while I didn't specify that the image should be pulled from docker.io and the image should have the :tilt-<hash> tag. (Details below)

Steps to Reproduce

I made a small repo to reproduce this issue. Below are the steps to reproduce the issue with the repo (I tested it on Windows).

  1. Clone or download this repo: https://github.com/Hiroki111/tilt-bug-fix
  2. Run a local Kubernetes cluster by Docker Desktop Kubernetes
  3. Make sure that kubectl configured to talk to that cluster
  4. Install Helm (https://helm.sh/docs/intro/install/)
  5. Install Tilt
  6. Run tilt up in the root of the repo
  7. Open the UI and click "configserver-deployment-" resource on the left-hand side

You will see something like:

Tracking new pod rollout (configserver-deployment-699f9c9fc7-244pc):
     ┊ Scheduled       - <1s
     ┊ Initialized     - <1s
     ┃ Not Ready       - (ContainersNotReady): containers with unready status: [configserver]
[event: pod configserver-deployment-699f9c9fc7-244pc] Pulling image "hiroki111/easycar-configserver:tilt-91fd32bba7608441"
[event: pod configserver-deployment-699f9c9fc7-244pc] Failed to pull image "hiroki111/easycar-configserver:tilt-91fd32bba7608441": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/hiroki111/easycar-configserver:tilt-91fd32bba7608441": failed to resolve reference "docker.io/hiroki111/easycar-configserver:tilt-91fd32bba7608441": docker.io/hiroki111/easycar-configserver:tilt-91fd32bba7608441: not found
[event: pod configserver-deployment-699f9c9fc7-244pc] Error: ErrImagePull
[event: pod configserver-deployment-699f9c9fc7-244pc] Error: ImagePullBackOff

In this log, I can see that Tilt is trying to pull an image from docker.io/user-name/easycar-configserver:tilt-91fd32bba7608441. However, there is actually no such image in docker.io.

I've pushed this hiroki111/easycar-configserver image to Docker Hub previously (Here's a link), but it doesn't have a tag like tilt-91fd32bba7608441. I feel that this could be the cause (i.e., Tilt is trying to pull an image with this tag from docker.io, while I didn't specify where the remote repository is located.)

Out of curiosity, in Tiltfile, I replaced the following code:

docker_build(
	"hiroki111/easycar-configserver",

with this:

docker_build(
	"hiroki111/easycar-configserver:2.0.0",

... but I still see the same error: failed to resolve reference "docker.io/hiroki111/easycar-configserver:tilt-91fd32bba7608441".

Context

tilt doctor Output

$ tilt doctor
Tilt: v0.35.1, built 2025-08-27
System: windows-amd64
---
Docker
- Host: npipe:////./pipe/dockerDesktopLinuxEngine
- Server Version: 28.3.0
- API Version: 1.51
- Builder: 2
- Compose Version: v2.38.1-desktop.1
---
Kubernetes
- Env: docker-desktop
- Context: docker-desktop
- Cluster Name: docker-desktop
- Namespace: default
- Container Runtime: containerd
- Version: v1.31.1
- Cluster Local Registry: none
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.
---
Analytics Settings
--> (These results reflect your personal opt in/out status and may be overridden by an `analytics_settings` call in your Tiltfile)
- User Mode: opt-out
- Machine: 995e124b54edc010081153ce6c69f192
- Repo: xuTwqamvGhuJHwWZJE+Fjg==

About Your Use Case

I've been trying Tilt to see if I can use it for developing microservice apps on my machine

Hiroki111 avatar Sep 04 '25 08:09 Hiroki111

thanks for the report! investigating...

nicks avatar Sep 05 '25 20:09 nicks

hmmm...i'm not able to reproduce this.

here's what's weird: tilt isn't involved here -- this should be all internal to Docker Desktop kubernetes. i.e., tilt will build the image to Docker Desktop's image store, and Docker Desktop's kubernetes will use it directly from the same image store -- no pulling required.

two questions:

  • have you tried using the tilt example projects? do they work for you, or do you get the same error?
  • have you tried resetting the Docker Desktop kubernetes cluster? i wonder if it got into a weird state.

nicks avatar Sep 05 '25 21:09 nicks

ah, I was able to reproduce this. I'm pretty sure this is a Docker Desktop Kubernetes footgun. Not sure if it's a bug or not (cc @landism do you know what the expected behavior is here?)

Repro steps:

  • Disable Docker Desktop containerd image store
  • Enable Docker Desktop kind

And you will get a kubernetes cluster where the image stores are disconnected.

I'd personally be fine with making this a Tilt error if you misconfigured Docker Desktop, but maybe we should just disallow this configuration in Docker Desktop

For now, the simplest workaround is to enable containerd (on Settings -> General) (assuming this is the same issue you're having)

nicks avatar Sep 05 '25 21:09 nicks

Yeah, this is a bug in Docker Desktop that has cost me too many hours. The GUI says that kind requires containerd but it's not really enforced. I plan to change Desktop to actually enforce it.

On Fri, Sep 5, 2025, 2:20 PM Nick Santos @.***> wrote:

nicks left a comment (tilt-dev/tilt#6608) https://github.com/tilt-dev/tilt/issues/6608#issuecomment-3259780704

ah, I was able to reproduce this. I'm pretty sure this is a Docker Desktop Kubernetes footgun. Not sure if it's a bug or not (cc @landism https://github.com/landism do you know what the expected behavior is here?)

Repro steps:

  • Disable Docker Desktop containerd image store
  • Enable Docker Desktop kind

And you will get a kubernetes cluster where the image stores are disconnected.

I'd personally be fine with making this a Tilt error if you misconfigured Docker Desktop, but maybe we should just disallow this configuration in Docker Desktop

For now, the simplest workaround is to enable containerd (on Settings -> General) (assuming this is the same issue you're having)

— Reply to this email directly, view it on GitHub https://github.com/tilt-dev/tilt/issues/6608#issuecomment-3259780704, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABY32J2LSJRNTRLYDJJYFUL3RH5CPAVCNFSM6AAAAACFTDMLF2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTENJZG44DANZQGQ . You are receiving this because you were mentioned.Message ID: @.***>

landism avatar Sep 06 '25 00:09 landism

@nicks @landism

Thanks for your replies.

have you tried using the tilt example projects? do they work for you, or do you get the same error?

No I haven't

have you tried resetting the Docker Desktop kubernetes cluster? i wonder if it got into a weird state.

No I haven't either

For now, the simplest workaround is to enable containerd (on Settings -> General) (assuming this is the same issue you're having)

Ok, let me try this approach. I'll write about what happens if I do this. For now, I'll keep this issue as it is.

Hiroki111 avatar Sep 08 '25 05:09 Hiroki111