Support for ARM64 architecture in Bitnami container images

[carrodher]

Currently, the Bitnami container images do not support the ARM64 architecture.

We are aware of the growing interest in this architecture and there are ongoing internal plans to release the Bitnami Community Catalog for ARM64 in the future, definitely, it is something we would like to support, but we need to find the bandwidth to do it in a proper way maintaining our quality standards.

At the moment, only our base image bitnami/minideb has support for ARM64 thanks to the community contribution in this PR: bitnami/minideb#90

Although there are some initiatives and Engineering work going on, we cannot guarantee a specific ETA for this topic. We will update this issue with more information.

Thanks for using Bitnami Containers and Helm Charts!

[samip5]

Are most of your builds being bult on either Travis CI or CircleCI?

Eg, how is the MariaDB container being built? I cannot find build files for it? (https://github.com/bitnami/bitnami-docker-mariadb)

[carrodher]

Are most of your builds being built on either Travis CI or CircleCI?

Eg, how is the MariaDB container being built? I cannot find build files for it? (https://github.com/bitnami/bitnami-docker-mariadb)

There is an internal pipeline based on Jenkins where we build the image with the docker CLI and then run several tests using the image in a deployment based on docker-compose and also in k8s via Helm Charts, you can find more information about this test and release process in the following link: https://docs.bitnami.com/tutorials/bitnami-best-practices-hardening-charts/#bitnami-release-process-and-tests.

Once tested the image, is published in different repositories such as

$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED        SIZE
public.ecr.aws/bitnami/apache      2.4       3156fbf7dfbd   23 hours ago   176MB
quay.io/bitnami/apache             2.4       3156fbf7dfbd   23 hours ago   176MB
docker.io/bitnami/apache           2.4       3156fbf7dfbd   23 hours ago   176MB
gcr.io/bitnami-containers/apache   2.4       3156fbf7dfbd   23 hours ago   176MB

[samip5]

Are most of your builds being built on either Travis CI or CircleCI? Eg, how is the MariaDB container being built? I cannot find build files for it? (https://github.com/bitnami/bitnami-docker-mariadb)

There is an internal pipeline based on Jenkins where we build the image with the docker CLI and then run several tests using the image in a deployment based on docker-compose and also in k8s via Helm Charts, you can find more information about this test and release process in the following link: https://docs.bitnami.com/tutorials/bitnami-best-practices-hardening-charts/#bitnami-release-process-and-tests.

Once tested the image, is published in different repositories such as

$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED        SIZE
public.ecr.aws/bitnami/apache      2.4       3156fbf7dfbd   23 hours ago   176MB
quay.io/bitnami/apache             2.4       3156fbf7dfbd   23 hours ago   176MB
docker.io/bitnami/apache           2.4       3156fbf7dfbd   23 hours ago   176MB
gcr.io/bitnami-containers/apache   2.4       3156fbf7dfbd   23 hours ago   176MB

Where is the Docker version used defined? What would need to be PR'ed to be able to produce multi-arch images with Docker Buildx?

[carrodher]

Are most of your builds being built on either Travis CI or CircleCI? Eg, how is the MariaDB container being built? I cannot find build files for it? (https://github.com/bitnami/bitnami-docker-mariadb)

There is an internal pipeline based on Jenkins where we build the image with the docker CLI and then run several tests using the image in a deployment based on docker-compose and also in k8s via Helm Charts, you can find more information about this test and release process in the following link: https://docs.bitnami.com/tutorials/bitnami-best-practices-hardening-charts/#bitnami-release-process-and-tests. Once tested the image, is published in different repositories such as

$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED        SIZE
public.ecr.aws/bitnami/apache      2.4       3156fbf7dfbd   23 hours ago   176MB
quay.io/bitnami/apache             2.4       3156fbf7dfbd   23 hours ago   176MB
docker.io/bitnami/apache           2.4       3156fbf7dfbd   23 hours ago   176MB
gcr.io/bitnami-containers/apache   2.4       3156fbf7dfbd   23 hours ago   176MB

Where is the Docker version used defined? What would need to be PR'ed to be able to produce multi-arch images with Docker Buildx?

I'm sorry but this part of the test & release pipeline is not public. You can find the Dockerfile for all the assets present in the catalog by looking into the different https://github.com/bitnami/bitnami-docker-ASSET repositories. As the Dockerfile is public you can build the image using them. In the same repository, you can customize the bash scripts used for the initialization logic.

In some of the above-mentioned issues, there are some community members that tried to customize some repositories to make them compatible with different architectures.

[samip5]

Are most of your builds being built on either Travis CI or CircleCI? Eg, how is the MariaDB container being built? I cannot find build files for it? (https://github.com/bitnami/bitnami-docker-mariadb)

There is an internal pipeline based on Jenkins where we build the image with the docker CLI and then run several tests using the image in a deployment based on docker-compose and also in k8s via Helm Charts, you can find more information about this test and release process in the following link: https://docs.bitnami.com/tutorials/bitnami-best-practices-hardening-charts/#bitnami-release-process-and-tests. Once tested the image, is published in different repositories such as

$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED        SIZE
public.ecr.aws/bitnami/apache      2.4       3156fbf7dfbd   23 hours ago   176MB
quay.io/bitnami/apache             2.4       3156fbf7dfbd   23 hours ago   176MB
docker.io/bitnami/apache           2.4       3156fbf7dfbd   23 hours ago   176MB
gcr.io/bitnami-containers/apache   2.4       3156fbf7dfbd   23 hours ago   176MB

Where is the Docker version used defined? What would need to be PR'ed to be able to produce multi-arch images with Docker Buildx?

I'm sorry but this part of the test & release pipeline is not public. You can find the Dockerfile for all the assets present in the catalog by looking into the different https://github.com/bitnami/bitnami-docker-ASSET repositories. As the Dockerfile is public you can build the image using them. In the same repository, you can customize the bash scripts used for the initialization logic.

In some of the above-mentioned issues, there are some community members that tried to customize some repositories to make them compatible with different architectures.

Okey so you kind of say that the community could help, but without the ability to see all of the build steps, there's pretty much NO WAY for people outside the org to help properly for example in the mariadb container build process to get it multi arched.

I cannot help without knowning how it's supposed to work under CI, as just changing docker files is not going to be nowhere enough to get this really anywhere. docker buildx requires a greater than 19.x, cannot remember.

What does this need to get this bumped to a higher priority in your roadmap?

[carrodher]

Are most of your builds being built on either Travis CI or CircleCI? Eg, how is the MariaDB container being built? I cannot find build files for it? (https://github.com/bitnami/bitnami-docker-mariadb)

There is an internal pipeline based on Jenkins where we build the image with the docker CLI and then run several tests using the image in a deployment based on docker-compose and also in k8s via Helm Charts, you can find more information about this test and release process in the following link: https://docs.bitnami.com/tutorials/bitnami-best-practices-hardening-charts/#bitnami-release-process-and-tests. Once tested the image, is published in different repositories such as

$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED        SIZE
public.ecr.aws/bitnami/apache      2.4       3156fbf7dfbd   23 hours ago   176MB
quay.io/bitnami/apache             2.4       3156fbf7dfbd   23 hours ago   176MB
docker.io/bitnami/apache           2.4       3156fbf7dfbd   23 hours ago   176MB
gcr.io/bitnami-containers/apache   2.4       3156fbf7dfbd   23 hours ago   176MB

Where is the Docker version used defined? What would need to be PR'ed to be able to produce multi-arch images with Docker Buildx?

I'm sorry but this part of the test & release pipeline is not public. You can find the Dockerfile for all the assets present in the catalog by looking into the different https://github.com/bitnami/bitnami-docker-ASSET repositories. As the Dockerfile is public you can build the image using them. In the same repository, you can customize the bash scripts used for the initialization logic. In some of the above-mentioned issues, there are some community members that tried to customize some repositories to make them compatible with different architectures.

Okey so you kind of say that the community could help, but without the ability to see all of the build steps, there's pretty much NO WAY for people outside the org to help properly for example in the mariadb container build process to get it multi arched.

I cannot help without knowning how it's supposed to work under CI, as just changing docker files is not going to be nowhere enough to get this really anywhere. docker buildx requires a greater than 19.x, cannot remember.

There are some parts public like the Dockerfile, bash initialization logic, etc so users can build their own image customizing it. What is not public is the internal logic we use to compile the source code if needed, build, test, and release the Bitnami container images.

What does this need to get this bumped to a higher priority in your roadmap?

The internal roadmap is based on business and product decisions, taking into account community users but other variables as well.

[samip5]

Where is the Docker version used defined? What would need to be PR'ed to be able to produce multi-arch images with Docker Buildx?

I'm sorry but this part of the test & release pipeline is not public. You can find the Dockerfile for all the assets present in the catalog by looking into the different https://github.com/bitnami/bitnami-docker-ASSET repositories. As the Dockerfile is public you can build the image using them. In the same repository, you can customize the bash scripts used for the initialization logic. In some of the above-mentioned issues, there are some community members that tried to customize some repositories to make them compatible with different architectures.

Okey so you kind of say that the community could help, but without the ability to see all of the build steps, there's pretty much NO WAY for people outside the org to help properly for example in the mariadb container build process to get it multi arched. I cannot help without knowning how it's supposed to work under CI, as just changing docker files is not going to be nowhere enough to get this really anywhere. docker buildx requires a greater than 19.x, cannot remember.

There are some parts public like the Dockerfile, bash initialization logic, etc so users can build their own image customizing it. What is not public is the internal logic we use to compile the source code if needed, build, test, and release the Bitnami container images.

Can you share any information about the whole process? Jenkins processes the Dockerfiles, is Jenkins build pipeline architecture aware as in does it support multiple architecture building via Docker Buildx?

What does this need to get this bumped to a higher priority in your roadmap?

The internal roadmap is based on business and product decisions, taking into account community users but other variables as well.

Which doesn't say much, and I don't feel like you actually answered my question so please do. Does one need to throw money at you before you take this as a higher priority than what it currently is?

[carrodher]

Where is the Docker version used defined? What would need to be PR'ed to be able to produce multi-arch images with Docker Buildx?

I'm sorry but this part of the test & release pipeline is not public. You can find the Dockerfile for all the assets present in the catalog by looking into the different https://github.com/bitnami/bitnami-docker-ASSET repositories. As the Dockerfile is public you can build the image using them. In the same repository, you can customize the bash scripts used for the initialization logic. In some of the above-mentioned issues, there are some community members that tried to customize some repositories to make them compatible with different architectures.

Okey so you kind of say that the community could help, but without the ability to see all of the build steps, there's pretty much NO WAY for people outside the org to help properly for example in the mariadb container build process to get it multi arched. I cannot help without knowning how it's supposed to work under CI, as just changing docker files is not going to be nowhere enough to get this really anywhere. docker buildx requires a greater than 19.x, cannot remember.

There are some parts public like the Dockerfile, bash initialization logic, etc so users can build their own image customizing it. What is not public is the internal logic we use to compile the source code if needed, build, test, and release the Bitnami container images.

Can you share any information about the whole process? Jenkins processes the Dockerfiles, is Jenkins build pipeline architecture aware as in does it support multiple architecture building via Docker Buildx?

In a generic way, the internal pipeline is tracking the upstream repositories, when there is a new version in any of the tracked applications, the source code is downloaded and compiled, then the Dockerfile and the initialization logic is generated (those files are public), with those files the container image is built, before releasing it, the container image is tested as part of different docker-composes and the Helm chart; if everything works as expected the container image is pushed to the different repositories.

The logic of this pipeline is not public, users can contribute to the Helm Charts (this is 100% public) or the container initialization logic, Dockerfile, docker-compose.yaml, etc; but unfortunately not with the pipeline logic itself that is the one in charge of build the image

What does this need to get this bumped to a higher priority in your roadmap?

The internal roadmap is based on business and product decisions, taking into account community users but other variables as well.

Which doesn't say much, and I don't feel like you actually answered my question so please do. Does one need to throw money at you before you take this as a higher priority than what it currently is?

When defining the roadmap, different parameters are considered such as the capacity of the team; we are a small team and we can't address all the requests we receive such as adding new Helm Charts to the catalog, implement new features, fix existing bugs, or as in this case, support a new arch. We would like to address everything ASAP, but the capacity is not infinite so we need to prioritize some topics.

Here you can find more info about Tanzu Application Catalog (TAC), which is the paid version of the Bitnami Community Catalog (BCC). In terms of the arch, ARM64 is not supported in TAC nor BCC, but other distros such as CentOS, PhotonOS, or Ubuntu are supported in TAC while only Debian is supported in BCC.

[samip5]

Where is the Docker version used defined? What would need to be PR'ed to be able to produce multi-arch images with Docker Buildx?

I'm sorry but this part of the test & release pipeline is not public. You can find the Dockerfile for all the assets present in the catalog by looking into the different https://github.com/bitnami/bitnami-docker-ASSET repositories. As the Dockerfile is public you can build the image using them. In the same repository, you can customize the bash scripts used for the initialization logic. In some of the above-mentioned issues, there are some community members that tried to customize some repositories to make them compatible with different architectures.

Okey so you kind of say that the community could help, but without the ability to see all of the build steps, there's pretty much NO WAY for people outside the org to help properly for example in the mariadb container build process to get it multi arched. I cannot help without knowning how it's supposed to work under CI, as just changing docker files is not going to be nowhere enough to get this really anywhere. docker buildx requires a greater than 19.x, cannot remember.

There are some parts public like the Dockerfile, bash initialization logic, etc so users can build their own image customizing it. What is not public is the internal logic we use to compile the source code if needed, build, test, and release the Bitnami container images.

Can you share any information about the whole process? Jenkins processes the Dockerfiles, is Jenkins build pipeline architecture aware as in does it support multiple architecture building via Docker Buildx?

In a generic way, the internal pipeline is tracking the upstream repositories, when there is a new version in any of the tracked applications, the source code is downloaded and compiled, then the Dockerfile and the initialization logic is generated (those files are public), with those files the container image is built, before releasing it, the container image is tested as part of different docker-composes and the Helm chart; if everything works as expected the container image is pushed to the different repositories.

The logic of this pipeline is not public, users can contribute to the Helm Charts (this is 100% public) or the container initialization logic, Dockerfile, docker-compose.yaml, etc; but unfortunately not with the pipeline logic itself that is the one in charge of build the image

So basically the portion that would be required to have changes made to it for multi-architecture image building aka pipeline logic that's charge in the building of it, is not public. So that would probably warrant a note on anything Bitnami stating that multi-arch is not supported on most of the images. Currently every time I see Bitnami anything, I take the assumption that there's no multi-arch images, which make it not usable at all.

Can you say the reason why the pipeline logic itself is not public?

What does this need to get this bumped to a higher priority in your roadmap?

The internal roadmap is based on business and product decisions, taking into account community users but other variables as well.

Which doesn't say much, and I don't feel like you actually answered my question so please do. Does one need to throw money at you before you take this as a higher priority than what it currently is?

When defining the roadmap, different parameters are considered such as the capacity of the team; we are a small team and we can't address all the requests we receive such as adding new Helm Charts to the catalog, implement new features, fix existing bugs, or as in this case, support a new arch. We would like to address everything ASAP, but the capacity is not infinite so we need to prioritize some topics.

Here you can find more info about Tanzu Application Catalog (TAC), which is the paid version of the Bitnami Community Catalog (BCC). In terms of the arch, ARM64 is not supported in TAC nor BCC, but other distros such as CentOS, PhotonOS, or Ubuntu are supported in TAC while only Debian is supported in BCC.

Okey, so if I were a paying customer of yours, would that actually make a difference in terms of getting an issue fast tracked? One would hope so, but somehow I doubt it.

[anthr76]

Is there a good reason to re-build every single image and by extension not have compatibility with an upstream image when using the Bitnami charts?

[adm271828]

I'm sorry but this part of the test & release pipeline is not public. You can find the Dockerfile for all the assets present in the catalog by looking into the different https://github.com/bitnami/bitnami-docker-ASSET repositories. As the Dockerfile is public you can build the image using them. In the same repository, you can customize the bash scripts used for the initialization logic.

In some of the above-mentioned issues, there are some community members that tried to customize some repositories to make them compatible with different architectures.

Hi,

The point is the Dockerfiles can not be used as they fetch precompiled binaries (directives such as: RUN . /opt/bitnami/scripts/libcomponent.sh && component_unpack ...), that are not available for ARM architecture.

So, in order to use the Dockerfiles, one has to recreate the binary archives with the very same software components selected by bitnami, without knowing which ones and not knowing how they were compiled. This may be achieved with great motivation and time but, in practice, is a waste of time. When I started a project that needed ARM images, I believed that it woud not be (too) difficult to rebuild docker images compatible with bitnami's chart, but I gave up pretty soon.

I wonder if people/organisations that choose to make bitnami's Helm Chart as their official chart (probably because they feel those chart's quality is good and possibly other good reasons) even realize that they, in fact, restrict the use of the chart to architectures supported by bitnami.

Best regards.

[balusarakesh]

• has anyone gotten a chance to try and build a bitnami docker image with publicly available ARM64 packages instead of the ones compiled by bitnami? if yes, did it work? example like this...we notice that we are downloading a package from bitnami servers....if we replaced that with the publicly available package, would that work?
• we are kind of in a hurry to migrate to ARM64 and we'd like to know if there is any alternative

Thank you

[samip5]

• we are kind of in a hurry to migrate to ARM64 and we'd like to know if there is any alternative

Thank you

Altirnative is to not use Bitnami images on arm64.

[adm271828]

• has anyone gotten a chance to try and build a bitnami docker image with publicly available ARM64 packages instead of the ones compiled by bitnami? if yes, did it work? example like this...we notice that we are downloading a package from bitnami servers....if we replaced that with the publicly available package, would that work?
• we are kind of in a hurry to migrate to ARM64 and we'd like to know if there is any alternative

Thank you

As far as I understood, bitnami packages have nothing to do with publicly available packages. They are the result of bitnami's secret build process and have their own structure. Packages enumerated in bitnami_components.json are downloaded with the corresponding lines in Dockerfile (RUN . /opt/bitnami/scripts/libcomponent.sh && component_unpack ...).

Some charts are said to be compatible with official docker images. I tried it with postgresql and was indeed able to deploy and boot the official ARM64 docker image with bitnami's chart.

However the "compatibility" is extremely limited and missleading. Almost every more elaborated options in the chart (that make bitnami's chart worth) were simply not working because the docker image was not compatible in the first place.

So if you started a projet with bitnami's charts and images, you're probably stuck to architectures selected by bitnami

Best regards.

[julweber]

+1 It would be so great to be able to use the bitnami helm charts on my raspberry pi

[turowicz]

@carrodher is anything planned for this?

[carrodher]

We are making some progress on different internal tools to allow the automated pipeline to compile and build images based on the new architecture, but there is still a way to go. We will update this thread once there are some visible progress

[stevefan1999-personal]

@carrodher Bitnami's ARM support is very crucial to AWS Graviton/Oracle Cloud Ampere platform for ARM Kubernetes to function well. Right now I can't even source a working Postgres cluster image in ARM K8S and have to run in lone database mode, so please let us know if there is any assistance needed from the community

[carrodher]

Thanks for the information @stevefan1999-personal, we are aware of the importance that ARM support has for certain scenarios and use cases, that's why we are working on this task. Unfortunately, the changes required should be done in our internal test & release pipeline and this is something not publicly available, thanks for your offer. Due to we are a small team and we are also working on different projects, the bandwidth dedicated to this topic makes us advance more slowly but we are doing some progress and, for sure, it is also we have in mind and we are working on.

[nischi]

@carrodher

Thank you very much that you working on this topic, we are looking forward for kafka support for M1 Chips. Do you already see any light in the end of the tunnel? Do you know by when we can use the arm images?

Thank you and kind regards

[carrodher]

There is some progress in the internal tools we use to build, test and release the container images, but there is still some work to do. Unfortunately, I can't provide an ETA at this moment since it depends on other priorities.

[nischi]

Oh bad to hear. Hope it will come soon.

[turowicz]

@carrodher is there any point for me to try and build the redis dockerfile for ARM? It seems that its using the minideb which has an ARM variant already.

[carrodher]

Unfortunately, it's not an option at this moment. Although all the initialization logic is public in the container repositories and based on bash, the components are built using the above-mentioned tools (the ones we are modifying to support the new distro), and it is something that needs to be re-built with different modifications to support the new arch:

# Install required system packages and dependencies
RUN install_packages acl ca-certificates curl gzip libc6 libssl1.1 procps tar
RUN . /opt/bitnami/scripts/libcomponent.sh && component_unpack "wait-for-port" "1.0.0-3" --checksum 7521d9a4f9e4e182bf32977e234026caa7b03759799868335bccb1edd8f8fd12
RUN . /opt/bitnami/scripts/libcomponent.sh && component_unpack "redis" "6.2.6-1" --checksum 172caf7ebf69ba8105ec7f05150a7e341fdf117c9650cbcb9191919a523ada86
RUN . /opt/bitnami/scripts/libcomponent.sh && component_unpack "gosu" "1.14.0-0" --checksum 3e6fc37ca073b10a73a804d39c2f0c028947a1a596382a4f8ebe43dfbaa3a25e

As you rightly said, minideb already supports ARM since it was one of the first pieces we worked on, but we need to continue working on other pieces/tools needed to build, test and release the containers.

[turowicz]

I never realised how very few good templates are out there, because I was only using Bitnami charts. Trying to find alternatives to work with ARM64 images from the arm64v8 project shows how necessary it is to get Bitnami helm charts to work on ARM. Helm without Bitnami is a complete mess.

[carrodher]

Some Helm charts support the use of non-bitnami images, I mean, you should be able to replace the image: block (registry, repository, tag, etc) to use another image based on ARM. Although this is not something tested nor supported out-of-the-box in all the charts, but you can give it a try in the meantime.

On our side, we understand the relevance of the issue and the wishes from the community, that's why we are actively working on it; but at this moment we can't spend more time per week than the one we are spending right now, that makes the process go slower than expected

[turowicz]

@carrodher the main worry is that changing the image will spin up the service, but all chart configuration may not be used. Currently I am looking for etcd and keycloak (with postgres) to run on ARM64 K3S instance.

[samip5]

@carrodher the main worry is that changing the image will spin up the service, but all chart configuration may not be used. Currently I am looking for etcd and keycloak (with postgres) to run on ARM64 K3S instance.

Postgres Bitnami chart supports the official Postgres image at least partially.

[XBeg9]

After reading this thread, I am going to stop using bitnami images in all my projects, that's not the way the opensource world should be, you hide the sources and lots of staff under the cover and remove the opportunity for the community to help with migration and finally can't give estimate on anything.

Anyone who will end up with Bitnami images, be aware, this is not true opensource project

[jamezrin]

After reading this thread, I am going to stop using bitnami images in all my projects, that's not the way the opensource world should be, you hide the sources and lots of staff under the cover and remove the opportunity for the community to help with migration and finally can't give estimate on anything.

Anyone who will end up with Bitnami images, be aware, this is not true opensource project

Better start asking for that refund