Build and push `tutorials` image to ECR

[heanhsok]

As discussed in https://github.com/causify-ai/helpers/issues/398#issuecomment-2743918431,

For the repository level,

We currently don't have a tutorials image built and stored in any registry, which means commands like invoke bash or invoke jupyter require building the image locally first.

It would be beneficial to have this image built and stored in a registry so it's sharable among us (through ECR) or even interns (through DockerHub), as there are use cases (for example, in LLM directory) where we might want to open a Jupyter notebook for quick experiments/scripting and install dependencies on the fly without needing to spin up a new runnable directory.

For subdirectory levels, we don't need to build and store any images. Interns/candidates can customize, build, run their own images locally as needed.

FYI @sonniki @dremdem @gpsaggese @samarth9008

[dremdem]

Based on the https://github.com/causify-ai/helpers/issues/398#issuecomment-2747868680 Can you pls build and push multi-arch tutorials image? FYI @sonniki

[samarth9008]

@heanhsok

Let us know if you need more guidance on how to do it. @dremdem can guide you?

[heanhsok]

Yes could you please share relevant doc on how to do it? @dremdem

We can use this flag to build the multi arch image right?

i docker_build_local_image --version 1.0.0 --multi-arch "linux/amd64,linux/arm64"

and how do we check if the image is multi arch?

[dremdem]

Yes could you please share relevant doc on how to do it? @dremdem

https://github.com/causify-ai/helpers/blob/master/docs/work_tools/all.devops_docker.how_to_guide.md#multi-architecture-build

We can use this flag to build the multi arch image right?

Right.

and how do we check if the image is multi arch?

We can use:

aws ecr batch-get-image \
    --repository-name tutorials \
    --image-id imageTag=dev \
    --accepted-media-types "application/vnd.docker.distribution.manifest.v1+json" \
    --output json --region eu-north-1 --profile ck | grep architecture

And in the output should be:

\"architecture\": \"amd64\"

it means for Linux

and

\"architecture\": \"arm64\",\n

it means for mac.

Let me know if something unclear, I'll happy to explain.

[heanhsok]

And in the output should be:

\"architecture\": \"amd64\"

it means for Linux

and

\"architecture\": \"arm64\",\n

it means for mac.

Got it. and if it's multi arch, both are shown right?

[dremdem]

Got it. and if it's multi arch, both are shown right?

Right.

You can check it for the cmamp:dev like:

aws ecr batch-get-image \
    --repository-name cmamp \
    --image-id imageTag=dev \
    --accepted-media-types "application/vnd.docker.distribution.manifest.v1+json" \
    --output json --region eu-north-1 --profile ck | grep architecture

[samarth9008]

Just FYI: As soon as you build the multi-arch image, they will be pushed automatically on the ECR registry. From the UI as well, you can check the archs.

@dremdem Feel free to take a lead on build the local image. LMK if you need any help when building and releasing the dev image.

[heanhsok]

I have built and pushed the multi arch version now. could you please check again? @dremdem

[dremdem]

I have built and pushed the multi arch version now. could you please check again? @dremdem

@heanhsok

Ok, thanks, it's looking good. The next step is to push it to the Dockerhub.

Can you pls run this:

i docker_tag_push_multi_build_local_image_as_dev --version 1.0.0 --target-registry dockerhub.causify

Feel free to reach out to me if something is not clear.

[heanhsok]

> i docker_tag_push_multi_build_local_image_as_dev --version 1.0.0 --target-registry dockerhub.causify
17:45:03 - INFO  hdbg.py init_logger:1018                               > cmd='/data/heanhs/src/venv/client_venv.helpers/bin/invoke docker_tag_push_multi_build_local_image_as_dev --version 1.0.0 --target-registry dockerhub.causify'
# docker_tag_push_multi_build_local_image_as_dev: version='1.0.0', local_base_image='', target_registry='dockerhub.causify', container_dir_name='.'
# docker_login: target_registry='dockerhub.causify'
17:45:03 - INFO  lib_tasks_docker.py _docker_login_dockerhub:324        Logging in to the target registry
  ... WARNING! Using --password via the CLI is insecure. Use --password-stdin.
  ... Login Succeeded
17:45:05 - INFO  lib_tasks_docker_release.py docker_tag_push_multi_build_local_image_as_dev:429 Pushing the local dev image 623860924167.dkr.ecr.eu-north-1.amazonaws.com/tutorials:local-heanhs-1.0.0 to the target_registry dockerhub.causify
docker buildx imagetools create -t causify/tutorials:dev-1.0.0 623860924167.dkr.ecr.eu-north-1.amazonaws.com/tutorials:local-heanhs-1.0.0
#1 [internal] pushing docker.io/causify/tutorials:dev-1.0.0
#1 0.000 copying sha256:229fe96739338218454485f9741cbab2f2a49e98a55e3ee8a97cf501fd2987c8 from 623860924167.dkr.ecr.eu-north-1.amazonaws.com/tutorials:local-heanhs-1.0.0 to docker.io/causify/tutorials:dev-1.0.0
#1 77.03 pushing sha256:229fe96739338218454485f9741cbab2f2a49e98a55e3ee8a97cf501fd2987c8 to docker.io/causify/tutorials:dev-1.0.0
#1 DONE 77.2s
docker buildx imagetools create -t causify/tutorials:dev 623860924167.dkr.ecr.eu-north-1.amazonaws.com/tutorials:local-heanhs-1.0.0
#1 [internal] pushing docker.io/causify/tutorials:dev
#1 0.000 copying sha256:229fe96739338218454485f9741cbab2f2a49e98a55e3ee8a97cf501fd2987c8 from 623860924167.dkr.ecr.eu-north-1.amazonaws.com/tutorials:local-heanhs-1.0.0 to docker.io/causify/tutorials:dev
#1 26.19 pushing sha256:229fe96739338218454485f9741cbab2f2a49e98a55e3ee8a97cf501fd2987c8 to docker.io/causify/tutorials:dev
#1 DONE 26.4s

Done. It ran sucessfully.

[samarth9008]

@dremdem

GJ in taking the lead on the issue.

[gpsaggese]

Good stuff.

1. As always, what was not clear from the documentation? Let's update / document it
2. The tutorials (will) contain many images, since there will be one per directory and we will have 10s-100s of dirs
   • We don't want to push all of them
   • I'd say we want to make sure the flow works by pushing the top one and one of the runnable dirs

Does it make sense? The specs for this repo are still in flux.

[heanhsok]

I'd say we want to make sure the flow works by pushing the top one and one of the runnable dirs

Yes which of the runnable dirs do you think is good for pushing to the registry to test the flow?

[gpsaggese]

I'd do tutorial_neo4j since @Prahar08modi and @tkpratardan are working on that so it should be working.

We can push only to Docker registry and maybe GitHub when we start unit testing the code (although we'll think about the pytest flow with several runnable dirs soon)

[sonniki]

I'd do tutorial_neo4j since @Prahar08modi and @tkpratardan are working on that so it should be working.

Might not be the best option -- for some reason, the poetry file in tutorial_neo4j contains a bunch of deps that shouldn't be there, like black and flake8. I mentioned this in https://github.com/causify-ai/tutorials/issues/54#issuecomment-2703883615. I'd propose tutorial_langchain, if you don't mind @gpsaggese

[samarth9008]

@heanhsok

Is it done?

[heanhsok]

Yes it was done.

• The image tutorials was built and pushed to ECR and Dockerhub.
• The release flow was also tested for tutorial_langchain.

[heanhsok]

Closing as done.