Windows-Containers

Windows-Containers copied to clipboard

Describe the bug

COPY commands have been working normally for years with our lts2019 windows container build on Agones, but started failing today with an error message of:

Dockerfile.windows:18
--------------------
  16 |     FROM mcr.microsoft.com/windows/servercore:${WINDOWS_VERSION}
  17 |
  18 | >>> COPY ./bin/sdk-server.windows.amd64.exe /agones/sdk-server.exe
  19 |     COPY ./bin/LICENSES ./bin/dependencies-src.tgz /agones/
  20 |
--------------------
ERROR: failed to solve: failed to compute cache key: mount callback failed on /tmp/containerd-mount3356703247: link /tmp/containerd-mount3356703247/Windows/INF/basicrender.inf /tmp/containerd-mount3356703247/Windows/System32/DriverStore/FileRepository/basicrender.inf_amd64_efdc64af60c69a6d/basicrender.inf: no such file or directory
make: *** [Makefile:619: build-agones-sdk-image-windows-ltsc2019] Error 1

Where ${WINDOWS_VERSION} is ltsc2019

To Reproduce

To reproduce this, just try copying something in:

ARG WINDOWS_VERSION=ltsc2019
FROM mcr.microsoft.com/windows/servercore:${WINDOWS_VERSION}

COPY ./emptyfile /emptyfile

(Assuming you have a buildx builder already)

❯ touch emptyfile
❯ docker buildx build --platform windows/amd64 --builder windows-builder-ltsc2019 --tag=windows-test .
[+] Building 18.6s (6/6) FINISHED                                                                                                                                                                                                                                                                                                              docker-container:windows-builder-ltsc2019
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                                0.1s
 => => transferring dockerfile: 156B                                                                                                                                                                                                                                                                                                                                                0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/servercore:ltsc2019                                                                                                                                                                                                                                                                                                      0.1s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                     0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                   0.0s
 => => transferring context: 28B                                                                                                                                                                                                                                                                                                                                                    0.0s
 => [1/2] FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:3c97a5c1c32ddb346c190f00a588da6e682a9a8160869f4969edfd7c6e4d1c03                                                                                                                                                                                                                                               18.3s
 => => resolve mcr.microsoft.com/windows/servercore:ltsc2019@sha256:3c97a5c1c32ddb346c190f00a588da6e682a9a8160869f4969edfd7c6e4d1c03                                                                                                                                                                                                                                                0.0s
 => => extracting sha256:0dd0445527a5079720e935502b31de927b8e22e5ca358026cf0bc8845c5ba5ce                                                                                                                                                                                                                                                                                          18.3s
 => ERROR [2/2] COPY ./emptyfile /emptyfile                                                                                                                                                                                                                                                                                                                                         0.0s
------
 > [2/2] COPY ./emptyfile /emptyfile:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:4
--------------------
   2 |     FROM mcr.microsoft.com/windows/servercore:${WINDOWS_VERSION}
   3 |
   4 | >>> COPY ./emptyfile /emptyfile
   5 |
--------------------
ERROR: failed to solve: failed to compute cache key: mount callback failed on /tmp/containerd-mount1556330329: link /tmp/containerd-mount1556330329/Windows/INF/basicrender.inf /tmp/containerd-mount1556330329/Windows/System32/DriverStore/FileRepository/basicrender.inf_amd64_efdc64af60c69a6d/basicrender.inf: no such file or directory

Expected behavior

The image should build 😃

Configuration:

• Edition: ltsc2019
• Base Image being used: mcr.microsoft.com/windows/servercore
• Container engine: docker, Running on Linux (debian).
• Container Engine version both 24.x and 26.x

Additional context

You can see a full build output from the Agones build pipeline here: https://console.cloud.google.com/cloud-build/builds/70b984e2-132b-4d1a-915a-862cb03f4830;step=16?e=13803378&project=agones-images

Using the previous SHA of @sha256:6fdf140282a2f809dae9b13fe441635867f0a27c33a438771673b8da8f3348a4 worked.

Also possibly worth noting ltsc2022, works still. We tested it!

which image is Using the previous SHA of @sha256:6fdf140282a2f809dae9b13fe441635867f0a27c33a438771673b8da8f3348a4 worked. in reference too?

which image is Using the previous SHA of @sha256:6fdf140282a2f809dae9b13fe441635867f0a27c33a438771673b8da8f3348a4 worked. in reference too?

mcr.microsoft.com/windows/servercore:ltsc2019

For reference, we have this PR now in place to unblock CI: https://github.com/googleforgames/agones/pull/3829

Tried it myself on a build node I have been using for years, though I have updated docker buildx since:

docker buildx version
github.com/docker/buildx v0.12.1 30feaa1a915b869ebc2eea6328624b49facd4bfb

I did use this version before KubeCon, and I did use mcr.microsoft.com/windows/servercore:ltsc2019 as a base image for the presentation where I talked about building Windows images. So, I think the image is the issue (Microsoft publishes a new image monthly).

This looks to be an issue with the image patches released Yesterday (May 14th). Work around is use April's patch images as done in https://github.com/microsoft/Windows-Containers/issues/493#issuecomment-2112937642

/cc @akarshm

Adding link to the slack discussion where we narrowed it down to the patch release https://kubernetes.slack.com/archives/C0SJ4AFB7/p1715733067064539

/cc @profnandaa

UPDATE: we've been investigating this issue, a few things worth noting:

1. The issue not actually with COPY but it is happening at the tail-end of FROM and so the reporting just estimates to the next line on the Dockerfile.
2. The issue is to do with extraction of the latest delta layer on ltsc2019:

   => extracting sha256:0dd0445527a5079720e935502b31de927b8e22e5ca358026cf0bc8845c5ba5ce 
   

   There's only one "offending" file Windows/INF/basicrender.inf (and its hardlinks); it so happens that on the TAR header (metadata), it's referenced as lowercase basicrender.inf, when the actual file in the layer is BasicRender.inf. Since Windows is not case-sensitive when it comes to files, this issue is only evident when the link operation is done on Linux (which is case-sensitive). Therefore, it reports as file-not-found. RCA is still going on to fix the issue.

PS. will be nice to re-tittle the issue to FROM/layer-extraction on ltsc2019 fails: "mount callback failed"

Hello, is there any plan / timeline to fix Windows images to be usable on Linux? We as Kubernetes community cannot release our images based on the mcr.microsoft.com/windows/servercore:ltsc2019@latest tag. It's not super critical, at least now, but you never know when a CVE comes and we will need to release images immediately.

@jsafrane -- a fix is currently going through validation; will update here once it's released.

hi, @profnandaa, can I gently ask you about a rough ETA for a fix?

The updated images released as part of the July 2024 security update today include the fix for this issue.

Sure, this is now fixed:

#5 [1/3] FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647
#5 resolve mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647 0.0s done
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 15.73MB / 573.97MB 0.2s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 54.53MB / 573.97MB 0.5s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 93.32MB / 573.97MB 0.8s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 131.07MB / 573.97MB 1.1s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 168.82MB / 573.97MB 1.4s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 206.57MB / 573.97MB 1.7s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 244.32MB / 573.97MB 2.0s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 284.16MB / 573.97MB 2.3s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 322.96MB / 573.97MB 2.6s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 362.81MB / 573.97MB 2.9s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 401.60MB / 573.97MB 3.2s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 442.50MB / 573.97MB 3.5s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 480.25MB / 573.97MB 3.8s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 519.05MB / 573.97MB 4.1s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 559.94MB / 573.97MB 4.4s
#5 sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 573.97MB / 573.97MB 5.2s done
#5 extracting sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e
#5 extracting sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 62.4s done
#5 DONE 67.6s

Guards have been put in place to prevent similar issues from happening in future.

Anyone else can ACK and we can proceed to close the issue.

I might be doing something wrong, but 41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647 does not work for me.

Dockerfile.reproduce:

FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647
RUN echo hello world > file

Build output:

docker build -f Dockerfile.reproduce  --platform=windows/amd64 .
[+] Building 458.1s (4/5)                                                                                                                                                                                                                                                                                      docker:default
 => [internal] load build definition from Dockerfile.reproduce                                                                                                                                                                                                                                                           0.0s
 => => transferring dockerfile: 202B                                                                                                                                                                                                                                                                                     0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                          0.0s
 => ERROR [1/2] FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647                                                                                                                                                                             457.9s
 => => resolve mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647                                                                                                                                                                                     0.0s
 => => sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647 375B / 375B                                                                                                                                                                                                                               0.0s
 => => sha256:f44fdada7eb2e71f80c964b2759421f4f7645a2df97a12aa82f66a8d50410414 596B / 596B                                                                                                                                                                                                                               0.0s
 => => sha256:b51a50ce03f12744163cde05017bec7149d704963fe41aed4b4a78472a98a3f8 788B / 788B                                                                                                                                                                                                                               0.0s
 => => sha256:cb524f6f22159378ea820d234d80ca09b79c2f0cc91315eeef11904e3ff36a21 1.60GB / 1.60GB                                                                                                                                                                                                                         410.6s
 => => sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 573.97MB / 573.97MB                                                                                                                                                                                                                     219.4s
 => => extracting sha256:cb524f6f22159378ea820d234d80ca09b79c2f0cc91315eeef11904e3ff36a21                                                                                                                                                                                                                               38.1s
 => => extracting sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e                                                                                                                                                                                                                                8.5s
------
 > [1/2] FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647:
------
WARNING: current commit information was not captured by the build: git was not found in the system: exec: "git": executable file not found in $PATH
Dockerfile.reproduce:1
--------------------
   1 | >>> FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647
   2 |     
   3 |     
--------------------
ERROR: failed to solve: failed to register layer: link /Files/Program Files/common files/Microsoft Shared/Ink/en-US/micaut.dll.mui /Files/Program Files (x86)/common files/Microsoft Shared/ink/en-US/micaut.dll.mui: no such file or directory

Fedora 39, with docker-ce-27.0.3-1.fc39.x86_64.

I tried plain mcr.microsoft.com/windows/servercore:ltsc2019, the same result (and the same SHA).

@jsafrane I got the same error as you when trying building Windows images with docker build on MacOS.

Try using docker buildx instead:

docker buildx build -f Dockerfile.reproduce --platform windows/amd64 .

docker buildx build -f Dockerfile.reproduce --platform windows/amd64 .

Same error as docker build.

I should have looked more closely your Dockerfile.reproduce: you can't use RUN instructions if you want to build Windows images on another host than Windows. (See for example https://stackoverflow.com/a/71910784/4074148)

Example of a working Windows Dockerfile built on MacOS:

FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647
COPY README.md README.md

Docker did not even reach RUN, it's stuck at unpacking layers from FROM. It fails with COPY too.

sorry about that, reopening to investigate.

It must be some issue in my setup. I can see that Kubernetes is able to build Windows images on Linux based on 41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647 and that's what matters to me (logs, search for the SHA there). It was failing with the old image.

/close

Thanks, @jsafrane. Closing this issue.