Add-ons are built differently between Home Assistant Operating System and Supervised

[felipecrs]

Describe the issue you are experiencing

When using Home Assistant Operating System, there is no docker buildkit and therefore it will never be used when building images. See:

• https://github.com/home-assistant/operating-system/issues/3935

When building add-ons using Supervised, it usually has docker buildkit because it is part of a normal docker installation process.

This means Supervisor produces different images depending on the underlying installation method.

This is a problem for add-on developers since the build environment their Dockerfile's are built is not consistent, and there are several differences between building with docker buildkit or not.

What type of installation are you running?

Home Assistant Supervised

Which operating system are you running on?

Debian

Steps to reproduce the issue

It is self-explanatory.

Anything in the Supervisor logs that might be useful for us?

N/A

System information

N/A

Supervisor diagnostics

No response

Additional information

My suggested fix is to always start the docker build process with the DOCKER_BUILDKIT=0 environment variable. It should be as simple as that.

[felipecrs]

@sairon just following up the conversation from https://github.com/home-assistant/operating-system/issues/3935 (which got locked), if HAOS used BuildKit, this issue would also be resolved:

• https://github.com/home-assistant/supervisor/issues/3223

I realized it here.

[agners]

Hm, good point, I guess we would prefer to make the two behave the same.

As discussed already in https://github.com/home-assistant/operating-system/issues/3935#issuecomment-2761167096, it's not really the plan to add buildkit to Home Assistant OS at this point. For now, the way to go would be to adjust the default daemon.conf to disable buildkit on Supervised... Or we just accept that difference 🤷

I realized it here.

Hm, I see so your actual issue here is that users are keep piling up Docker images Superviser pulled during build because they were the base image back then.

What we could do maybe is creating a Fixup which checks for unused images, and prune them from time to time (set auto to true to the Fixup).

In general, I think we'd rather move away from supporting building on the target at all. We've been playing with the idea to use a more lightweight container runtime, potentially without any build support.

GitHub offers container image storage as well as build pipelines using GitHub actions are quite straight forward, I'd rather suggest to go towards that direction. Frenck's community add-on are quite popular choice to template from.

[felipecrs]

For now, the way to go would be to adjust the default daemon.conf to disable buildkit on Supervised... Or we just accept that difference 🤷

I proposed a much simpler solution, which is to always invoke docker build command with DOCKER_BUILDKIT=0.

As discussed already in home-assistant/operating-system#3935 (comment), it's not really the plan to add buildkit to Home Assistant OS at this point.

I actually understood that buildx would not be added to the HAOS image, but it could be done through a sidecar container plugin like hassio_audio and hassio_dns. It could be called hassio_addon_builder, and maybe it would only be pulled/installed when the user first needs to build an add-on.

The legacy builder is deprecated for a long time and will be removed at some point. This change is inevitable, and building workarounds would just be losing time.

What we could do maybe is creating a Fixup which checks for unused images, and prune them from time to time (set auto to true to the Fixup).

That would certainly be better than nothing. But migrating to BuildKit would still be the best.

In general, I think we'd rather move away from supporting building on the target at all.

I hope you never do that. Sorry, but I respectfully think you didn't think this through.

Being able to build add-ons locally allows to quickly iterate when developing the add-on. It's a crucial feature for add-on developers.

Also, if you make add-on developers have to push 100% of their add-ons, you are raising the bar for people to develop new add-ons. In my opinion, it should be made as simple as possible for people to build and share their own add-ons. If building becomes a bottleneck for a certain add-on, it should be up to its users to request the developer to ship it prebuilt.

GitHub offers container image storage as well as build pipelines using GitHub actions are quite straight forward, I'd rather suggest to go towards that direction. Frenck's community add-on are quite popular choice to template from.

I don't think the template is a good solution.

It brings hundred lines of code to a project which the add-on developer has now to maintain.

For example, it is up to the add-on developer to maintain his copy of the template up to date with the template itself. That's a burden.

As an add-on developer, I want to avoid as much as possible losing control of my project, and also, I want to avoid having to maintain unnecessary code.

Some add-ons, like my Netdata add-on, is simply better served through local building. It basically only adds an entrypoint.sh on top of the existing Netdata official image, and then I don't need to think about:

• Wasting space in some container registry. Even if it's free for users, there are unnecessary costs for the registry host and the world/environment.
• Maintaining credentials for pushing
• Having to orchestrate the build and push of multiple architectures
• Users having to wait before the push is finished before they can update their add-ons
• Most importantly: having to maintain potentially hundred lines of code to make the build and push happen correctly even when it's for no good.

[felipecrs]

We've been playing with the idea to use a more lightweight container runtime, potentially without any build support.

A more lightweight container runtime will be a very welcome improvement to HAOS.

But there is no need to remove building functionality because of it. You can just delegate the building functionality to the sidecar add-on builder plugin.

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[felipecrs]

This issue is not stale.

[agners]

When building add-ons using Supervised, it usually has docker buildkit because it is part of a normal docker installation process.

I've tried to look into force using non-BuildKit builds. And it appears that is already the case:

My Supervised installation has a recent Docker version with buildx available:

$ docker info
Client: Docker Engine - Community
 Version:    28.2.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.24.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
...

Building a simple add-on which uses BuildKit feature:

# syntax=docker/dockerfile:1.4
FROM alpine
RUN --mount=type=cache,target=/root/.cache \
    apk add --no-cache curl

Leads to:

Note that we don't invoke a shell command to build images, instead we are using the docker-py Python library to call Docker directly:

https://github.com/home-assistant/supervisor/blob/0b79e09bc034db9395c91b0615af4b47932cd062/supervisor/docker/addon.py#L676-L678

From a quick look, it doesn't seem there is a way to force enable/disable BuildKit. In fact, it seems that docker-py does not support BuildKit: https://github.com/docker/docker-py/issues/2230

This means Supervisor produces different images depending on the underlying installation method.

What is the effective differences you are seeing? Maybe just having it installed (but not used) lead to diffferent results? Or maybe it is down to the Docker version actually, not BuildKit?

[felipecrs]

What is the effective differences you are seeing? Maybe just having it installed (but not used) lead to diffferent results? Or maybe it is down to the Docker version actually, not BuildKit?

Honestly, I don't remember. It has been a long time since I switched from Supervised to Home Assistant OS.

---

Can we convert this into a feature request to use buildx as a sidecar service like I mentioned here instead?

An alternative library could be used for that purpose: https://github.com/gabrieldemarmiesse/python-on-whales.

It looks like docker-py was mainly used by docker-compose v1 (Python), and given docker-compose is no longer Python, it seems to be in a hiatus.

[agners]

What is the effective differences you are seeing? Maybe just having it installed (but not used) lead to diffferent results? Or maybe it is down to the Docker version actually, not BuildKit?

Honestly, I don't remember. It has been a long time since I switched from Supervised to Home Assistant OS.

Can we convert this into a feature request to use buildx as a sidecar service like I mentioned here instead?

I agree, moving building capability into an add-on would be a nice way to deploy it only to systems where it is necessary, and deploy the same logic for all installation types.

We collect feature requests through discussions nowadays (and all this discussion is a good example why: Things often are more complicated and evolve, affect different/multiple repositories, issues are not that well suited for that). There is a category for Home Assistant OS and Supervisor: https://github.com/orgs/home-assistant/discussions/categories/home-assistant-os-supervisor. Can you create a discussion there?

It looks like docker-py was mainly used by docker-compose v1 (Python), and given docker-compose is no longer Python, it seems to be in a hiatus.

Yeah, unfortunately. But from what I can tell it is still the "best" Python library to interface with Docker.

[felipecrs]

Awesome. I will create a discussion there. And I will investigate it a little bit more.

[felipecrs]

For others arriving here, the discussion is: https://github.com/orgs/home-assistant/discussions/77