Support additional docker buildkit options

[dsiebel]

Currently BuildKit / buildx support is Skaffold seems to be rudimentary: When useBuildKit:true it simply sets env DOCKER_BUILDKIT=1, effectively enabling BuildKit but not allowing to specify any of the command line options that docker buildx build support, for example. This means that --cache-from with, for example type=registry can not be used without a custom build command like this one:

apiVersion: 'skaffold/v3'
kind: 'Config'
metadata:
  name: 'buildx-inline-caching'
build:
  artifacts:
    - image: us-docker.pkg.dev/skaffold-examples/buildx-inline-caching
      context: .
      custom:
        buildCommand: |
          echo
          echo "Building image '${IMAGE}' for platforms '${PLATFORMS}' from context '${BUILD_CONTEXT}'"
          echo "Image is expected to exist remotely: ${PUSH_IMAGE}"
          echo
          args=()
          if [[ "${PUSH_IMAGE}" == true ]]; then
            args+=("--push")
          elif [[ "${PLATFORMS}" != *","* ]]; then
            args+=("--load")
          fi
          docker buildx build \
            --file "${BUILD_CONTEXT}/Dockerfile" \
            --tag "${IMAGE}" \
            --cache-to "type=inline" \
            --cache-from "type=registry,ref=us-docker.pkg.dev/skaffold-examples/buildx-inline-caching:cache" \
            --platform "${PLATFORMS}" \
            "${args[@]}" \
            "${BUILD_CONTEXT}"

This is what we currently use. Skaffold is being invoked with --tag cache after a successful build, which re-tags and pushes the image with the :cache tag so it can be used as a cache source in subsequent builds. However this approach doesn't really scale well, having to implement this custom script across potentially hundreds of skaffold.yaml files and repositories. It would be way nicer if the buildx build arguments would be supported by Skaffold natively.

As a start, I imagine using the combination of the useBuildKit and cacheFrom docker builder config options to construct the --cache-from option for example and adding a cacheTo config option for the docker builder to do the same for the --cache-to option. This way one could leverage the advanced caching mechanisms of BuildKit through Skaffold. Replacing the call to docker build with docker buildx build should already be transparent once you installed buildx (it is on my machine), but I need to double check.

Let me know what you think! I am happy to provide a first implementation, but I could definitely use some helping getting the tests right.