Support bake file as build target

[jalaziz]

Description

Today, a docker compose file can service as a bake file. This is great when you already use a docker compose file for your build spec, but docker bake allows for much more robust and complex build configurations than what compose supports.

Now that Bake is GA, it would be great if we could instead specify a bake file and target as the build spec for a compose service. This would allow for much richer build definitions that are shared across development and production, while still allowing existing docker compose workflows to run as is.

Put another way, it would be wonderful if docker compose up --build could invoke a Bake build rather than only being able to reference a standard Dockerfile.

[ndeloof]

What do you have in mind as "more robust and complex build configurations" when it comes to a single image being built ? AFAICT docker compose build definition gives you all the features bake supports, and actually latest Docker Compose release can rely on bake under the hood to run the build, which make multi-services sharing common resources build more efficient.

[jalaziz]

What do you have in mind as "more robust and complex build configurations" when it comes to a single image being built ?

Yes. Perhaps the most powerful feature of bake is being able to coordinate builds across dependencies. For example, with Bake, it's trivial to have a base image that is used in your final images and have it built automatically as needed. It's also trivial to use inheritance and targets to share a Dockerfile between a production build and debug-enabled dev build. Finally, Docker Bake supports functions.

AFAIK, none of what I've listed is supported by the docker compose build configuration without an extra layer of coordination.

[hanikesn]

I'd like to add that I also use matrices to easily define dozens of targets for microservices sharing a common go build step including cross platform builds and properly configured caching. Bake files enable many advanced use cases for mono repos. Being able to reuse those targets from a docker-compose filed would be immensely helpful.

[ndeloof]

I can hardly imagine how compose could run bake build while still managing multiple services declared in compose.yaml - especially as our middle terms plan is to remove native build support from compose and delegate builds to bake, generating a bake.json config from compose declaration.

The only option I can imagine is we support https://github.com/docker/compose/issues/12376

IMHO if you heavily rely on a bakefile to optimize your build, you should just run docker buildx bake before you run compose commands.

Closing as "not planned"

[hanikesn]

IMHO if you heavily rely on a bakefile to optimize your build, you should just run docker buildx bake before you run compose commands.

The nice part of having compose itself support it is the integration in 3rd party tooling, e.g. launching a devcontainer or similar. That way you wouldn't have to instruct users and it'd be possible to use them in Github Codespaces as well.

#12376 would take care of that as well of course.

Thanks for your consideration!

[antiBaconMachine]

AFAICT docker compose build definition gives you all the features bake supports

It mostly does this if you are prepared to replicate your entire existing bake target into the compose file and flatten any inheritence whilst your are at it.

I'm not actually sure I can see a way to do build dependencies if you did this either e.g. have one target depend on a base image build.

IMO docker compose is a great general purpose orchestration system, bake is a great build tool. I think the more specific tool should be consumed by the more general tool.

Right now it works the other way around, bake can consume compose files and generate targets from them. Given the bake definitions in compose files are somewhat more limited than a full external bake definition (no inheritence, have to flatten and duplicate everywhere) this all seems suboptimal.

I get that compose has always been a fully declarative source of truth, it feels odd to delegate some of it's work to a completely different declarative system and there a probably a hundred other issues I haven't thought about.

Personal data point from my experience: The way we use these systems is

• Bake is the source of truth for production images
• Compose is used for developer orchestration

So currently we end up with developer images which are defined similarly in x-bake sections but not quite the same as our actual production images in a docker-bake.hcl file. It looks very attractive to us to have docker compose use the existing bake targets from our bake files, not have to specify them again in compose.

[cideM]

I'd like to add a somewhat concrete use case for this: for local development we use the --mount=type=ssh feature so that docker build can leverage the host's SSH agent to e.g., authenticate with private GitHub repositories. This is setup now so that docker buildx bake Just Works™ (and it was a lot of fun to setup -- I really like Bake).

We also have a Docker Compose file that starts the service I alluded to above. I expected that docker compose up --build would utilize the Bake configuration and therefore use SSH agent forwarding. But the build started with docker compose up --build fails with an SSH authentication error, whereas docker buildx bake works.

[ndeloof]

@cideM see https://github.com/compose-spec/compose-spec/blob/main/build.md#ssh

[cideM]

I know about that and am using it but now I’m duplicating this exact same configuration between Bake and Compose

[ndeloof]

Let's keep this one opened to collect feedback and integration proposals

[Veetaha]

I'd like to be able to define my build with a bake file, and reference it from docker-compose, so that docker-compose invokes bake to perform thebuild. Isn't it as simple as that?

I just don't want to re-define my build configuration in docker-compose which is guaranteed to be much worse and potentially impossible;

[ndeloof]

I wonder this could be addressed using a top-level images attribute as suggested by https://github.com/compose-spec/compose-spec/issues/188

[jalaziz]

I wonder this could be addressed using a top-level images attribute as suggested by compose-spec/compose-spec#188

It could be, but the problem would still b the same. We'd want to be able to reference a bake file and a target, rather than making the compose spec be the source of truth for the build.

#188 seems to be about sharing build configuration across services. This issue is about enabling a separate Docker Bake file to be the source of truth for builds.

[segevfiner]

Docker compose literally seems to use bake as default build backend... And bake reads compose.yml and translates it as well, It would make sense to be able to be able keep build definitions in a docker-bake.hcl and compose details in compose.yml, and just have docker compose build know about such a setup and build using the definitions in docker-bake.hcl as well. And there is another feature missing from compose in that you can't tell it to not start certain services that are only used as bases for build.

This kinda seems silly at this point, why leave gaps like that between the two tools? I'd bet its some purist saying compose's scope is not to handle building images, despite having the build section which is often used, and despite integrating with bake and having bake read build definitions from compose.yml files. So we make things harder for users but stay pure to our arbitrarily defined scope.

[glours]

Hey @segevfiner Just a friendly reminder not to make assumptions about internal priorities or timelines, we’re often juggling a lot, and some things simply haven’t been tackled yet, not intentionally left out.

If this feature is critical for you, we’d genuinely welcome a PR, just be sure to handle edge cases carefully and maintain backward compatibility.

Let’s keep the conversation focused and constructive. Thanks!

[segevfiner]

Not complaining about your priorities or timelines. It's an open source project and you don't owe me anything.

What I meant was the long discussions about such things that went on and on in other closed issues unfortunately. Which seems to be why we don't have some way to add a service in compose to only be used as additional context without compose starting it. Which could have been useful for my use case, but all issues about it have been closed with claims as I mentioned. Which probably should be simpler then integrating compose and bake more tightly as requested here, yet due to supposed simplicity of such a thing leads to such arguments and often no action (AKA Bike shedding).