Bake; placeholders or templating targets?

[ciaranmcnulty]

I have a job that builds a matrix of PHP versions, base distros (and some other stuff IRL). I want to combine it into one bake, but still source the versions from env vars. I'm not sure if there's a way to do this in bake, or if I need to generate HCL/JSON separately?

I could generate the default group pretty easily if I set an env var like PHP_VERSIONS=7,8 DISTROS =alpine,ubuntu:

variable PHP_VERSIONS {}
variable DISTROS {}

group "default" {
  targets = flatten(
    [for d in split(",", "${DISTROS}"):
      [for v in split(",", "${PHP_VERSIONS}"):
        "${d}-php-${v}"
      ]
    ]
  )
}

However I can't see how to avoid explicitly enumerating all the targets:

target "alpine-php-8" {}
[...etc...]

Ideally I'd have some way to template the targets, is there anything possible in bake's HCL?

[ciaranmcnulty]

Incidentally now . isn't allowed in target names this makes things more complicated!

[tonistiigi]

Related from #buildkit slack

[tonistiigi]

cc @cpuguy83

[ciaranmcnulty]

Thanks @tonistiigi that sort of approach would work well

[ciaranmcnulty]

A fairly simplistic placeholder approach without the matrix being implemented in bake would actually enable a lot of stuff, not necessarily just matrix builds.

target "foo-${bar}" {
   [... variable bar is defined in this scope ...]
}

This wouldn't be a backwards compatibility issue, as template sequences are not allowed ye in target names. If it looks too much like interpolation, a different syntax could be used.

My use case above would then look like this:

variable PHP_VERSIONS {}
variable DISTROS {}

group "default" {
  targets = flatten(
    [for d in split(",", "${DISTROS}"):
      [for v in split(",", "${PHP_VERSIONS}"):
        "${d}-php-${v}"
      ]
    ]
  )
}

target "${flavour}-php-${version}" {
    dockerfile = "${flavour}/Dockerfile"
    args = {
        PHP_VERSION = "${version}"
    }
}

The 'apply to all targets' in #1149 would now be simple. When I build foo with this config, the tags would be applied (multiple same-name targets already get merged so here both blocks would match)

target "foo" {...}

target "${targetName}" {
    tags: buildTags("$targetName");
}

It would also cover the case in #956, although the merge order will matter:

target "foo" {
    target = "bar" # this would be overridden
}

target "${targetName}" {
    target = "${targetName}"
}

target "bar" {
    target = "bar" # this would be used
}

I'm sure there are many more applications

[tonistiigi]

@ciaranmcnulty This is ambiguous.

target "${flavour}-php-${version}" {
    dockerfile = "${flavour}/Dockerfile"
    args = {
        PHP_VERSION = "${version}"
    }
}

target "${foo}-php-${bar}${baz}" {
    dockerfile = "Dockerfile"
    args = {
        FOO_VERSION = "${version}"
    }
}

target "some-php-${bar}" {
    dockerfile = "Dockerfile"
    args = {
        SOME_VERSION = "${version}"
    }
}

What is getting built?

[ciaranmcnulty]

@tonistiigi great question! I think you probably didn't meant to keep ${version} in all of them?

target "${flavour}-php-${version}" {
    dockerfile = "${flavour}/Dockerfile"
    args = {
        PHP_VERSION = "${version}"
    }
}

target "${foo}-php-${bar}${baz}" {
    dockerfile = "Dockerfile"
    args = {
        FOO_VERSION = "${bar}${baz}"
    }
}

target "some-php-${bar}" {
    dockerfile = "Dockerfile"
    args = {
        SOME_VERSION = "${bar}"
    }
}

Taking an input like some-php-11 then I'd probably match the placeholders greedily:

target "some-php-11" { # flavour='some', version='11'
    dockerfile = "some/Dockerfile"
    args = {
        PHP_VERSION = "11"
    }
}

target "some-php-11" { # foo='some', bar='11', baz=''
    dockerfile = "Dockerfile"
    args = {
        FOO_VERSION = "11"
    }
}

target "some-php-11" { # bar='11'
    dockerfile = "Dockerfile"
    args = {
        SOME_VERSION = "11"
    }
}

Currently if you have multiple targets of the same name they get merged, with later ones taking priority. So I'd expect the final state to be something like this

target "some-php-11" {
    dockerfile = "Dockerfile"
    args = {
        PHP_VERSION = "11",
        FOO_VERSION = "11",
        SOME_VERSION = "11"
    }
}