docker_build or docker_build_with_restart with same images but with different commands

[zkutasi]

We have an Kubernetes Deployment that contains two containers, they are derived from the same Dockerfile, but they have two entrypoints, two main() functions. We use the first as an init-container and the second for the main application.

Now when I issue something like this:

docker_build_with_restart(
    'IMAGENAME',
    entrypoint=[...],
)

... since with the docker_build_with_restart() entrypoint is singular AND mandatory, I cannot differentiate between the init container and the main container, Tilt replaces both in my Deployment YAML descriptor, making the init container to "fail", actually it does not execute the proper code at all, and the main container fails to start up. If I use docker_build() instead, and omit the entrypoint parameter completely things work as each image will keep its defined entrypoint, but I lose the live update capability of Tilt if I am correct.

What I want to do: Propagate my initcontainer unmodified into the Deployment while allow live update on my main container, although both of them come from the same image. In other words, only selectively replace the entrypoint with the given one. I would imagine, I could control this with the image names: specify a map of entrypoints on the image names. Or if it is possible, even give an option to not even override the entrypoint on a given image name.

What do you think? What are my other options currently?

[lizzthabet]

we've heard a couple similar use-cases outlined in #3385. others have created workarounds using custom_build, and there's an example in that issue linked above. have you tried working with a custom build yet? let me know if you have questions about that approach!

ps. feel free to also +1 the other issue! it helps us gauge interest and prioritize what we work on.

[zkutasi]

Thanks for the tip...

However I think that WA is not OK for this scenario. Honestly after reading 10 times the mentioned WA, I still do not get it fully. I think the problem I am having is that I cannot modify the container image name in the manifest... in that WA I do not know how that could work if the manifest has the exact same image names.

What the documentation states:

https://docs.tilt.dev/api.html#api.custom_build

ref ( str ) – name for this image (e.g. ‘myproj/backend’ or ‘myregistry/myproj/backend’). If this image will be used in a k8s resource(s), this ref must match the spec.container.image param for that resource(s).

What I do not get is that the spec.container.image is. it is spec.containers... and it is a list in Kubernetes. also it seems to also match the spec.initContainers as well. I would clarify this a bit in the documentation first. But to me this means that if I have a deployment YAML, and in it two containers that run the same image, both of them will be replaced with the same content. This is perfect until one wants to add live updates... which is where this gets to be a problem.

What I would do is to add another parameter to all of the builders, lets say containerNames, where I could say WHICH image to replace, based on the container's name in the manifest. To make it even more robust, I would also add a separate parameter for initContainerNames. Both of them could be lists. For backward compatibility, the default behavior when these are not set is to replace everything. if they are set, then the replacement is selective, based on the name of the containers.

Also I really would like to use docker_build_with_restart, and I have seen there is a cousin for custom_build_with_restart. If you have any clue what can I do to make these work. let me know.

What WA I am thinking now is to have my helm chart copied to a temp-dir, then run a sed on it to replace the init-container image name and give it another text content and then going from here.

[zkutasi]

Meanwhile I have found a solution: I think I followed what you both suggested, but for me it was a bit hard at first :)

I have split the docker_build_with_restart() into a custom_build() and a custom_build_with_restart(), using unique image names... and I already had my helm chart completely custom-templated and parsed: I handle each and every object individually and act on them, edit them and sort them into groups. Now I added a rewrite of the image names to the two new ones, and when now I add them into their own k8s_resource(), Tilt picks up the new images, and for the main image it does with the one utilizing live updates, and the init container without.

One extra thing I had to sort out is that I used the ability to specify a custom Dockerfile... and these new functions do not support that easily, but I ended up using the STDIN capability of docker build. All in all it might work, but I am too tired to try it out today :). Thanks anyway for pointing me towards these custom builds.

[lizzthabet]

@zkutasi, i'm glad you were able to figure out a setup that works for you! i get that the custom builds can be difficult to configure and your feedback is useful for helping us prioritize live_update + docker developments.

were you able to specify a specific Dockerfile with custom_build? i believe the docker build cli has -f option which should let you set one.

thanks again for your feedback!

[zkutasi]

Hi @lizzthabet , for reference and for help to others, I try to put here my solution... although my Tiltfiles are much more complicated than this, and full of tricks.

This is how I do the dockerfile... I need to have a custom one, as in production most of our Dockerfiles are securized, removing zypper and lots of things, like tar, so I put them back... and yes, I use docker build -f, but it was tricky:

tilt_dockerfile_contents = str(local("sed 's/zypper rm.*/echo/' ./" + project_dirs[0] + "/Dockerfile", quiet=True)).replace('$', '\\$')
custom_build('init',
    'docker build --tag $EXPECTED_REF --build-arg BASE_OS_VERSION=' + image_baseos_version + ' ' + project_dirs[0] + ' -f-<<EOF\n' + tilt_dockerfile_contents + '\nEOF',
    deps=[],
)

custom_build_with_restart('main',
    'docker build --tag $EXPECTED_REF --build-arg BASE_OS_VERSION=' + image_baseos_version + ' ' + project_dirs[0] + ' -f-<<EOF\n' + tilt_dockerfile_contents + '\nEOF',
    live_update=[
...
    ],
    deps=[
...
    ],
    entrypoint=["./entrypoint.sh"]
)

Then later on I do this:

    h = local('helm template ...', quiet=True)
    decoded = decode_yaml_stream(h)

    if decoded:
        decoded_keep = []
        for d in decoded:
...
                    # The main deployment contains two images and this is not handled by Tilt
                    # So above we enable live updates for the image, but we should not rewrite the entrypoint for both
                    # For this we do two custom builds above and rewite here the image names to later on match with those
                    if object_kind == 'Deployment' and object_name == 'xxx':
                        d['spec']['template']['spec']['initContainers'][0]['image'] = 'init'
                        for c in d['spec']['template']['spec']['containers']:
                            if c['name'] == 'xxxx':
                                c['image'] = 'main'

So I have in my Tiltfile full control over all the YAMLs, as I decode them one by one and assign them into lists and then from these lists I do the k8s_resource() calls... but I can do whatever I want with them, and I have like 12 tricks in here already, but it gives me absolute control.

[lizzthabet]

thanks, @zkutasi! i'm glad you were able to get your setup working.