custom_build(tag=) is only used to tag the local image, not the one pushed to the cluster

[fkaleo]

When using custom_build(..., tag=MYTAG), the image is tagged locally with :MYTAG but the image pushed to the kubernetes cluster has a different tag based on the beginning of the image id, for example :tilt-6838c54d01f71b92. This makes the tag feature less useful for people who need to know what will be the name of the image in the cluster.

[fkaleo]

I will try to workaround it pushing manually and using disable_push=True.

[fkaleo]

Actually I cannot do a manual push since I don't know what the expected image reference will be. It is $EXPECTED_REF only in the local docker registry.

In the end I do the following which results in 2 pushes with 2 different tags :tilt-tmp and :tilt-6838c54d01f71b92 to the registry the kubernetes cluster has access to (x2 kind load):

custom_build('myimage',
             'docker build -t $EXPECTED_REF . && kind load docker-image $EXPECTED_REF',
             deps=['.'], tag='tilt-tmp')

This is slow because kind load is not smart atm and always reuploads the image due to https://github.com/kubernetes-sigs/kind/issues/380

[nicks]

hmmmm...that's surprising to me.

Can you elaborate a bit more on why you need to know the name of the image in the cluster? What are you trying to use it for?

I ask because Kubernetes itself has bugs around reporting the name of the correct image in the cluster. So there are certain use-cases here that won't work on Kubernetes itself, regardless of what Tilt does (https://github.com/kubernetes/kubernetes/issues/51017)

PS -- are you using kind? are you aware that Tilt will use kind load natively if it detects you're pushing to a kind cluster?

[fkaleo]

Yeah it was surprising to me too but then internal/build/custom_builder.go seems pretty clear about it:

        dig := digest.Digest(inspect.ID)
        tag, err := digestAsTag(dig)

thus discarding expectedTag passed to ExecCustomBuilder.Build()

[fkaleo]

And yep I'm using kind. Tilt does kind load fine indeed but with the incorrect tag.

[fkaleo]

I'm going to checkout kubernetes/kubernetes#51017 though, it seems it could be related.

[nicks]

@jazzdan has more context on this than I do. He spent a lot of time experimenting with the exact semantics of custom_build.

AFAIR, the problem is that the tag needs to change whenever the contents of the image change. This ensures that Kubernetes picks it up correctly and deploys the right image. If you're depending on it deploying a pre-defined tag, weird things happen and you don't actually get the results you want.

Or is there a use-case that we're missing?

[nicks]

(for example, would it help if you could query Tilt to get the tag of the image it ended up deploying?)

[fkaleo]

Thanks @nicks for taking the time. So actually I don't care that much about the tag that Tilt used to deploy the image. However what I care about is that I can have the same image tagged with a custom tag of my chosing and that will be available from the kubernetes cluster. Concretely speaking, my Tiltfile line docker build -t $EXPECTED_REF . && kind load docker-image $EXPECTED_REF does the trick and it might be the right way to do it given how specific my case is. Unfortunately this is very slow due to https://github.com/kubernetes-sigs/kind/issues/380

We could stop there and say there is no bug in tilt, however the tag parameter in custom_build is slightly misleading. Its name and the documentation

the tag you expect the resulting image to have; we set $EXPECTED_REF=imagename:tag

really made me feel like that tag would be carried over to the image pushed to the kubernetes cluster.

[fkaleo]

(for example, would it help if you could query Tilt to get the tag of the image it ended up deploying?)

That would not unfortunately. I need to know the tag before Tilt comes into the picture hence I need to be able to decide it myself.

[nicks]

We'll update the documentation to make the EXPECTED_REF semantics a little clearer.

Unfortunately, both Tilt and Kubernetes really need the tag to change if the container contents change.

Have you considered using labels? That's usually the standard solution we see people use when they want to get their resource after it's deployed, and they want to use a pre-determined identifier.

[fkaleo]

You mean Kubernetes labels? In my case I need to select a Docker image, so labels wouldn't cut it I think.

[fkaleo]

As discussed with maia, perhaps an acceptable solution would be to push 2 different tags to the Kubernetes cluster: the one generated (hash of the image ID) and the one specified by the user in custom_build(tag=...)

[sivaprasadreddy]

I am also facing this issue. When I first run tilt up it builds the image with a tag ex: 'docker.io/library/dummy-service:tilt-build-1658198393' and while Loading image to KIND giving the following error:

Image: "docker.io/library/dummy-service:tilt-2208a9bc0457db24" with ID "sha256:2208a9bc0457db247d805d9863dec5545cf5e5ab15ebc859f8221d33dfaf8cc0" not yet present on node "mk8s-control-plane", loading...
ERROR: failed to load image: command "docker exec --privileged -i mk8s-control-plane ctr --namespace=k8s.io images import --snapshotter overlayfs -" failed with error: exit status 1
     
Command Output: unpacking docker.io/library/dummy-service:tilt-2208a9bc0457db24 (sha256:6510be109a1077533f190e84f9545017f693a74b81ac4020d8586b4d70ff8164)...ctr: content digest sha256:95129a5fe07e89c1898dc40a027b291d5fe33a67b35a88f0f0eaf51ea691f0b5: not found

On further changes it will rebuild the image, but trying to load the image with old SHA and giving the error as follows:

Successfully built image 'docker.io/library/dummy-service:tilt-build-1658198483'


BUILD SUCCESSFUL in 57s
5 actionable tasks: 1 executed, 4 up-to-date

STEP 3/4 — Pushing dummy-service:tilt-2208a9bc0457db24
     Loading image to KIND
     Image: "docker.io/library/dummy-service:tilt-2208a9bc0457db24" with ID "sha256:2208a9bc0457db247d805d9863dec5545cf5e5ab15ebc859f8221d33dfaf8cc0" found to be already present on all nodes.

STEP 4/4 — Deploying
     Applying YAML to cluster
     Objects applied to cluster:
       → dummy-service:deployment

     Step 1 - 0.03s (Force update)
     Step 2 - 57.79s (Building Custom Build: [dummy-service])
     Step 3 - 0.56s (Pushing dummy-service:tilt-2208a9bc0457db24)
     Step 4 - 0.14s (Deploying)
     DONE IN: 58.51s 


Tracking new pod rollout (dummy-service-5887959f98-xwvbp):
     ┊ Scheduled       - <1s
     ┊ Initialized     - <1s
     ┃ Not Ready       - (ContainersNotReady): containers with unready status: [dummy-service]
[event: pod dummy-service-5887959f98-xwvbp] Container image "dummy-service:tilt-2208a9bc0457db24" already present on machine
[event: pod dummy-service-5887959f98-xwvbp] Error: failed to create containerd container: error unpacking image: failed to resolve rootfs: content digest sha256:2208a9bc0457db247d805d9863dec5545cf5e5ab15ebc859f8221d33dfaf8cc0: not found

[jkremser]

I am hitting the same issue with k3s. Clearly, if disable_push=True then user needs to deliver the image somehow to the k8s cluster. In k3s it's the k3d image import $ref (~ kind load), however I can't do that because the real image that is going to be used is not accessible by any env var.

[nicks]

I'm going to close this issue -- in all cases where people had this problem, they were simply deeply confused on how things fit together. Our documentation is much better now.

@jkremser - Tilt works out of the box with k3d and kind, we've collaborated a lot with the kind and k3d teams to make it work well. You shouldn't need to use custom_build. See this guide: https://docs.tilt.dev/choosing_clusters.html

Or this guide: https://k3d.io/v5.2.1/usage/registries/#using-a-local-registry

Tilt and K3d will automatically communicate with each other about how to push to the registry.

[jkremser]

thank you, I will take a look

[AlmogBaku]

@nicks I am still confused. How can I push a specific tag to the registry?

In my use-case I have an app that deploys other containers as pods/jobs. To do that, I need to build and push the images to the registry. Unfortunately, I can't find a way to push an image with the "latest" tag at all.

Is there a way I can solve it without bounding myself to kind load? in other words- what's the native tilt way to achieve that?

[AlmogBaku]

Unfortunately, this is still an issue.

I wrote a workaround:

def discover_registry_local():
    cmd = "kubectl --context " + k8s_context() + " -n kube-public get cm local-registry-hosting --template  '{{ index .data \"localRegistryHosting.v1\" }}'"
    res = decode_yaml(local(cmd, quiet=True))
    return res['host']


registry_local = discover_registry_local()

registry_tag = registry_local + "/" + name + ":latest"
cmd = "docker build -t $EXPECTED_REF " + conn_dir + " && docker tag $EXPECTED_REF " + registry_tag + " && docker push " + registry_tag
print(cmd)
custom_build(name,
             cmd,
             [conn_dir],
             disable_push=True,
             )

This is very ugly. I expected it to be solved as part of this feature

[nicks]

I'm going to lock this issue to avoid further confusion.

I think when people try to do this, they're usually down an XY rabbit hole -- https://xyproblem.info/. i.e., they're trying to solve some other problem X, and think that if they could somehow reverse engineer tilt's image building pipeline they could fumble their way to a solution. And so they start trying to use custom build to reverse engineer the image pipeline.