Add a way to leverage external cache source in Docker export

[themightychris]

The new --multi-layer option for hab pkg export docker is really powerful, but it can't be taken advantage of in a CI environment where the Docker build cache isn't persisted between builds. Instead you end up with 90+ layer images that share nothing.

Docker has a means of using "external cache sources" that can solve this. It involves passing --build-arg BUILDKIT_INLINE_CACHE=1 to docker build to trigger some additional metadata being written into the image, and then next time you rebuild the same image in a new environment, you can docker pull the previous image and specify it via --cache-from during the next docker build.

Currently, there is no way to pass either of these options to the underlying docker build call when using hab pkg export docker --multi-layer. Please either add a way to pass additional arbitrary params to docker build (like HAB_DOCKER_OPTS does for studios) or add higher-level options for enabling an external caching workflow

I haven't been able to identify any downside of BUILDKIT_INLINE_CACHE, other than it only being available in newer engines. Given that --multi-layer is a newer feature behind an option, it might be worth considering having it set BUILDKIT_INLINE_CACHE by default since the whole point of the feature is to maximize cache efficiency, so its output would always include caching metadata. The user should be able to specify --cache-from manually as it will be the user's responsibility in a CI environment to pull a previous build into the local docker cache ahead of time, but it might also make sense for --multi-layer to apply some magic when it's not manually specified like assuming a :latest tag its about to update is a good cache source if it exists already

[themightychris]

I misunderstood some things initially, it appears that BUILDKIT_INLINE_CACHE is only necessary when Docker's new build engine BuildKit is enabled with DOCKER_BUILDKIT in the env.

So without buildkit being activated, we really only need a way to pass --cache-from through -- or a general path to append arbitrary docker build args

[themightychris]

Another note: it is significant that --cache-from can be passed multiple times

[themightychris]

Is this roughly the changes needed?

1. Add add_cache_arg around here: https://github.com/habitat-sh/habitat/blob/450fe85fea08fe8c37c0b9fdc22ac8a4b4bbf836/components/pkg-export-container/src/cli.rs#L21
2. Insert one or more --cache-from args around here: https://github.com/habitat-sh/habitat/blob/d5497169422a8c2dac275c1d3f79fab8b1a52996/components/pkg-export-container/src/engine/docker.rs#L55
3. Figure out if buildah has an equivalent, or bail if someone tries to use them together around here: https://github.com/habitat-sh/habitat/blob/d5497169422a8c2dac275c1d3f79fab8b1a52996/components/pkg-export-container/src/engine/buildah.rs#L54

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[themightychris]

This is still a major limitation of the feature that could be addressed with a simple passthrough flag

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.