Multiple levels of wildcards not supported or not working properly

[csMACnz]

I used to be able to copy all files of a specific type from two-levels nested. This no longer works in BuildKit.

Why? To avoid cache busting. We want to restore our dependencies from all *.csproj files, and not invalidate that cache layer when a source file changes.

see https://stackoverflow.com/a/58697043 as the defacto approach most dotnet docker builds will be using.

If we just copy everything over first, that means any change triggers cache-busting and a full package restore is again required. Without wildcards, we have to manually maintain a long list of csproj file COPYs that is error-prone and unmaintainable across many solutions.

# These all worked without buildkit but now fail with buildkit
COPY src/*/*.csproj ./
COPY */*/*.csproj ./

This puts all found files into a single folder (which at least works for the goal we are trying to achieve)

I now get errors like:

error from sender: readdir: open *: The filename, directory name, or volume label syntax is incorrect.

or sometimes nothing is copied, so i get missing files errors later in the next RUN command.

As far as I an tell, COPY */*.csproj ./ still works as expected, when csproj files are only one-level nested, but doesn't work for the two-level nested case.

As a workaround, I might be able to move to a one-level nested structure, though ideally i wouldn't have to change my structure to fit docker for something that used to work?

Thanks!

[pierregorce]

I'm having the same problem while configuring buildkit for same caching purpose in .NET. Is there an alternative ?

COPY */*.csproj ./ (with one-level nested) not working either, is there a tricks ?

Thanks 😊

[gvanbrakel]

I don't think it is about multiple level of wildcards. When I run the below script, that has only one level of wildcard, it fails:

mkdir project-x-dependencies

echo "line-of-text" > project-x-dependencies/file1.txt
	
echo FROM busybox > Dockerfile
echo COPY project-*-dependencies/file1.txt  /work/ >> Dockerfile

docker build .

It appears to me that it is about wildcards in directories.

[YoeriVD]

"needs investigation"

Any update on this? after the last update, no one in our team using the windows version of docker can build their images anymore :/

EDIT: Fellow googlers (temp) fix can be found here https://github.com/moby/moby/issues/42133#issuecomment-796804743

[sbabych]

Good morning, in our case we have smth like: COPY ./drivers/build/3rdparty-jdbc-drivers/**/* jars/3rdparty/ and with "buildkit": true it doesn't work

[qrjo]

If this is gonna be fixed, it would be nice have a way to have these wildcards and preserve the folder structure as well.

[ASHWINI-GUPTA]

Same issue happened while I was trying to build the - https://github.com/Squidex/squidex. After setting "buildkit": false was able to build but resetting "buildkit": true its giving an error from sender: readdir: open backend\extensions\*: The filename, directory name, or volume label syntax is incorrect..

[vchirikov]

Same with

COPY src/dotnet/*/*.csproj ./

doesn't work with buildkit, works fine without buildkit

[Schwaller]

Please fix this - it's creating so much work and can't be that difficult to solve this - please. IMHO this ticket doesn't need any investigation, it needs solving :D (talking of labels)

[jahsome]

Stumbled into a workaround thanks to a very unique combination of sheer laziness and force of stubborn will.

I just added a "staging" layer and slammed my entire context to copy from It appears to have the added benefit of standardizing to *nixy / paths from the get-go, and just copying in --from=ctx instead:

FROM scratch as ctx
WORKDIR /app
COPY . /app

FROM mcr.microsoft.com/dotnet/sdk:6.0 as build
COPY --from=ctx "app/src/*/*.csproj" "./"

I'm sure this is bloating my caches with unneeded builds though. I'm trying, but I can't figure out how to explicitly exclude the stage from caching (via Dockerfile at least).

[fadamsen]

Is there anything we can do to get this looked at?

[Cjewett]

Is there any hope if this issue being addressed?

[ghost]

Hi all This seems like a very strange issue that globbing does not work? Is this fo for buildkit on windows only (via WSL)

The proposed work arounds above defeat the purpose of layer caching. in docker build?

[cabeaulac]

This is for buildkit. It's happening to me on my Mac.

-Chad

On Wed, Jun 28, 2023 at 4:01 AM Piotr Kula @.***> wrote:

Hi all This seems like a very strange issue that globbing does not work? Is this fo for buildkit on windows only (via WSL)

The proposed work arounds above defeat the purpose of layer caching. in docker build?

— Reply to this email directly, view it on GitHub https://github.com/moby/buildkit/issues/1900#issuecomment-1611197821, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGPWHR2X4EJMJEAISHMBQ3XNQFG7ANCNFSM4U47NNVA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[ghost]

OK so its not a host issue.. its just globbing is not implemented in copy for buildkit

That is very strange to have feature parity missing like that.. especially when the legacy builder is about to be removed.

Who do we escalate this too because I have noticed buildkit issues have been open for years and nobody really doing anything here?

[andremarcondesteixeira]

Any updates on this?

[liven-maksim-infotecs]

Any plans about it?

[tonistiigi]

Tested this and seems to work fine. Can we close this?

» cat Dockerfile
from alpine
copy */*/*/*.csproj /out/
run find /out && stop

» find src
src
src/dotnet
src/dotnet/bb
src/dotnet/bb/dd.csproj
src/dotnet/bb/cc.csproj
src/dotnet/aa
src/dotnet/aa/bb.csproj
src/dotnet/aa/aa.csproj

 => [2/3] COPY */*/*/*.csproj /out/                                                                                       0.1s
 => ERROR [3/3] RUN find /out && stop                                                                                      0.2s
------
 > [3/3] RUN find /out && stop:
0.184 /out
0.185 /out/aa.csproj
0.185 /out/bb.csproj
0.185 /out/cc.csproj
0.185 /out/dd.csproj
0.185 /bin/sh: stop: not found

 => CACHED [2/3] COPY src/dotnet/*/*.csproj /out/                                                                          0.0s
 => ERROR [3/3] RUN find /out && stop                                                                                      0.2s
------
 > [3/3] RUN find /out && stop:
0.163 /out
0.164 /out/aa.csproj
0.164 /out/bb.csproj
0.164 /out/cc.csproj
0.164 /out/dd.csproj

If the intention is to keep parts of the original directory structure then use --parents. Examples in https://www.docker.com/blog/new-dockerfile-capabilities-v1-7-0/

** that also some comments mention works with --parents but has never been something supported by COPY before --parents was added in buildkit.