Windows-Containers
Windows-Containers copied to clipboard
microsoft
As a Developer I want to be able to build Windows Containers with BuildKit
It is possible to use buildx (which uses buildkit) if there are no RUN commands in the docker file:
docker buildx create --name img-builder --use
docker buildx build --no-cache --pull --push \
--platform "windows/amd64" -t $imagetag -f $dockerfile .
docker buildx rm img-builder
ARG nanoserverTag="mcr.microsoft.com/windows/nanoserver:1809-amd64"
# othertags list: curl -L https://mcr.microsoft.com/v2/windows/nanoserver/tags/list
FROM ${nanoserverTag}
COPY some.exe "/Windows/System32/some.exe"
It would be interesting to see if could use some windows emulator connected to Binfmt_misc as QEMU is used for other architectures
There is an issue logged on BuildKit repo https://github.com/moby/buildkit/issues/616
@jsturtevant If you wanted to go down that path, I believe Wine can be used with binfmt_misc, although I'm not sure how well that would work with namespaces since there's a WineServer and configuration directory to create, and you probably don't want that in the final image...
It would be easier to prototype that on a raw Linux box than via the Linux support in Docker on Windows, I think.
This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.
Current status of my work on https://github.com/moby/buildkit/issues/616 is that it's stalled waiting on a pile of 4 containerd PRs to be reviewed and advanced. The top of the stack is https://github.com/containerd/containerd/pull/4419, but there's some outstanding design questions on the top two PRs regarding HCS behaviours, see https://github.com/microsoft/hcsshim/issues/853, but the bottom two are just pending containerd maintainer review to merge or rework as appropriate.
And after that, will come fixing path-related issues in the BuildKit LLB layer, for which I'd only scratched the surface of in https://github.com/moby/buildkit/pull/1621, exploring the solution space, and which needs to be reworked per maintainer feedback.
This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.
Status unchanged, still waiting on merging or feedback on containerd support for Windows container creation/export/mount support, and then continuing the work on BuildKit itself, starting from https://github.com/moby/buildkit/pull/1621
This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.
This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.
Still working through containerd PRs per above. One of four has landed, and one needs to be rewritten or at least iterated upon further. I've also opened PR on hcsshim again to remove some duplication of code with the containerd PRs.
That said, Docker is looking at supporting containerd on Windows for 21.x (https://github.com/moby/moby/issues/41455), which might mean we get BuildKit on Docker+containerd even without that containerd PR stack, as the parts I was trying to implement in containerd (WCOW layer management) are provided by Docker when Docker uses containerd, as far as I know.
I haven't looked closely at how BuildKit integrates with Docker though, it's possible it'll still need fixes in containerd, and either way, BuildKit itself has higher-level issues to be resolved, e.g., https://github.com/moby/buildkit/pull/1621.
This issue has been open for 30 days with no updates. @immuzz, please provide an update or close this issue.
Not much has changed since December. The containerd PRs are now in a reasonable state, two are ready-to-go and one (https://github.com/containerd/containerd/pull/4419) is functional but possibly needs some discussion about the test suite changes, and depends on a hcsshim feature PR (https://github.com/microsoft/hcsshim/pull/901).
Some recent testing suggests that that hcsshim PR might have issues with Windows 10 20H2 that don't appear on Windows Server LTSC2019, but I haven't definitely proved that as I have only tested the latter via CI pipelines so far. Edit: Still using CI pipelines, I confirmed that the PR works on RS5 on AppVeyor (VS 2019 image), but fails on my personal Windows 10 20H2 box, suggesting that some part of the reverse engineering used to build that PR was incorrect or incomplete.
@weijuans-msft can you provide clarity here as part of your effort to provide containerd parity with Docker developer experience.
This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.
Compared to January, I've identified and have a fix for the issue that https://github.com/microsoft/hcsshim/pull/901 shows on Windows 10 20H2, but that PR is still pending merge, and then the fix will be up for PR.
The containerd PRs (https://github.com/containerd/containerd/pull/4399, https://github.com/containerd/containerd/pull/4415, and https://github.com/containerd/containerd/pull/4419) should all be in a reasonable state to flow through, although the latter depends on https://github.com/microsoft/hcsshim/pull/901, often triggers a now-known containerd issue (https://github.com/containerd/containerd/issues/4924) in its test suite, and also frequently triggers an unknown (hcsshim?) issue that I've yet been unable to replicate outside containerd.
It's possible that the Docker/containerd integration for Windows (https://github.com/moby/moby/issues/41455 is the best place to track that) will mean some or all of those PRs stop being critical-path for BuildKit on Windows, and then the BuildKit-local platform issues (i.e. filesystem handling and similar) can be addressed.
@TBBle great work here! I changed this to "In Progress" as I am seeing you and lots of others in the moby and containerd community working on this. Thank you.
This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.
This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.
This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.
Without this, is there any good way to handle build-time secrets for Windows containers?
One kind-of heavy-weight option, used by ue4-docker, is running a service outside the container that exposes the secret, and have the container image build pull that secret over the network connection.
You can see ue4-docker's implementation of the external host and a Git Credential Helper to query it, including a generated token to provide the bare-minimum protection.
A significant cost to this approach is that the token ends up in the image as a build-arg, so layer caching is confounded; I don't know if the same thing happens when using BuildKit secret mounts, but I assume not.
In ue4-docker's case, the container builds are done under control of the script, so the credential endpoint is served by the script, rather than being left running all the time, which make this approach more feasible than it would be for just a Dockerfile being distributed for use with docker build.
This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.
I'd love to assist, as I really need this feature to speed up the process of building Docker Windows images.
Is there any way to build stages in parallel in Windows without using BuildKit?
This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.
This issue has been open for 30 days with no updates. @weijuans-msft, please provide an update or close this issue.
I don't know if we can freeze issues. If anyone knows, let me know :).
I could use help to be enlightened on the benefit or the pain points this can address. I see these so far in this thread:
- containerd parity
- Run Linux and Windows container side by side
- Can use secrets
What else?
How does this help in your or your company's modernization effort? Is this really Developer centric or it can add value to your DevOps in production?
We would like this so that eventually docker buildx bake support can be added so that we can define one hcl file with all our image builds declaratively configuring things like image tags and args, along with parallel builds out of the box. We use this on linux right now and it works really well.
What else?
Off the top of my head, I'd bring up:
-
BuildKit automatically parallelizes multi-stage builds for you. It's probably my favorite feature. See: https://www.gasparevitta.com/posts/advanced-docker-multistage-parallel-build-buildkit/
-
If you manage a container build farm or CI system with many workers, BuildKit makes it easy to share Docker build caches across several machines, which can speed up similar builds across them. See: https://link.medium.com/VYWZJPwnVkb
-
Other projects exclusively dependant on BuildKit would finally support Windows, like the wonderful "Earthly", sort of Makefile Dockerfile hybrid built upon the BuildKit core APIs. See: https://earthly.dev (Only for Linux containers at this time due to the missing Windows BuildKit support.)
If I think of more I'll extend this list. 🤔
