buildkit
buildkit copied to clipboard
Docker CLI Windows and long paths (260 characters) not working
Hi there,
I think there is an issue with long paths files (longer than 260 characters), that are copied with COPY
command within a Dockerfile.
Its something like https://github.com/moby/moby/issues/15775, but now related to Buildkit.
Issue
This is what I did in Powershell:
# Create a dir
mkdir C:/project/aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL
# Create a file that is 278 chars long
echo "hello" > C:/project/aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL/278aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaaYaaaaaaaaaZaaaaaaaaaA.txt
# Create a file that is 258 chars long
echo "hello" > C:/project/aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL/258aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaa.txt
Having following Dockerfile
FROM alpine:3.9.4
COPY . /
RUN ls -la /aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL
Running without Buildkit both files are inside the image, as expected:
PS C:\project> Remove-Item env:DOCKER_BUILDKIT
PS C:\project> docker build --progress plain --no-cache .
Sending build context to Docker daemon 7.68kB
Step 1/3 : FROM alpine:3.9.4
---> 055936d39205
Step 2/3 : COPY . /
---> a289f56da592
Step 3/3 : RUN ls -la /aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL
---> Running in 847e5d70224f
total 16
drwxr-xr-x 2 root root 4096 Feb 17 17:37 .
drwxr-xr-x 1 root root 4096 Feb 17 17:53 ..
-rwxr-xr-x 1 root root 16 Feb 17 17:37 258aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaa.txt
-rwxr-xr-x 1 root root 16 Feb 17 17:35 278aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaaYaaaaaaaaaZaaaaaaaaaA.txt
Removing intermediate container 847e5d70224f
---> f3984890e8e9
Successfully built f3984890e8e9
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
Running with Buildkit only the shorter file is inside the image:
PS C:\project> $env:DOCKER_BUILDKIT=1
PS C:\project> docker build --progress plain --no-cache .
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 192B done
#1 DONE 0.4s
#2 [internal] load .dockerignore
#2 transferring context: 2B 0.0s done
#2 DONE 0.4s
#3 [internal] load metadata for docker.io/library/alpine:3.9.4
#3 DONE 0.0s
#4 [1/3] FROM docker.io/library/alpine:3.9.4
#4 CACHED
#5 [internal] load build context
#5 transferring context: 591B 0.0s done
#5 DONE 0.1s
#6 [2/3] COPY . /
#6 DONE 0.1s
#7 [3/3] RUN ls -la /aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaa...
#7 1.252 total 12
#7 1.252 drwxr-xr-x 2 root root 4096 Feb 17 17:38 .
#7 1.252 drwxr-xr-x 1 root root 4096 Feb 17 17:55 ..
#7 1.252 -rwxr-xr-x 1 root root 16 Feb 17 17:37 258aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaa.txt
#7 DONE 1.4s
#8 exporting to image
#8 exporting layers
#8 exporting layers 0.1s done
#8 writing image sha256:c739fecfe5648b356900207a6cb7d6672138d1ffc3dfdbf2f5251986eda451db
#8 writing image sha256:c739fecfe5648b356900207a6cb7d6672138d1ffc3dfdbf2f5251986eda451db 0.0s done
#8 DONE 0.2s
My Setup
Using current Docker Desktop for Windows:
PS C:\project> docker version
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:22:37 2019
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:29:19 2019
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
I just run into this issue and it was quite frustrating to find the root cause. In my situation some random files were missing in the container and the build would fail. Like the author describes this only happened when using BuildKit. I feel the only reason why not a lot of people have run into this is because nobody is using BuildKit on windows apparently.
Also ran into this issue... the fact that it's silently failing during copy makes it extremly hard to identify this about path being too long
Did you solve this issue? I notive for node container this issue.
Ran into the same issue with Java codebase.. took a while to figure out that the compilation was happening due to missing class files with long paths !
I too have run into this issue. Is this being looked into?
If it is not easy to fix: It would be nice, if there were an error message, complaining about too long paths
Failing silently isn't a great approach to being unable to read a file. It makes me question the integrity of the containers I build.
I worked around the problem by using the following path syntax:
docker build \\?\D:\path -f .
Mapping D:\path
to \\?\D:\path
allows the use of long file paths.
I too ran into the same problem and spent a few hours debugging an error message that I got at Docker run, complaining with a cryptic message that was only a side effect of this real problem. As people pointed out, silent failures definitely isn't great. Here's the link to my problem ,arising out of this , on stackoverflow: https://stackoverflow.com/questions/72604450/mlflow-load-model-fails-python/72613045#72613045
Cross-linking -> https://github.com/moby/buildkit/issues/1086
@tonistiigi -- also if possible, could we add an extra tag on top of area/windows
to differentiate LCOW vs. Windows Containers?
/cc. @gabriel-samfira
Looks like this issue was resolved along the way?
Can't repro it no more. On Docker v24.0.6 (that runs Buildkit backend for Linux by default):
PS C:\dev\play\buildkit\long_path> docker build -t lcow_longpath . --progress plain --no-cache
....
#7 [3/3] RUN ls -la /aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL
#7 0.302 total 12
#7 0.302 drwxr-xr-x 2 root root 4096 Oct 19 08:52 .
#7 0.302 drwxr-xr-x 1 root root 4096 Oct 19 08:58 ..
#7 0.302 -rwxr-xr-x 1 root root 16 Oct 19 08:55 278aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaaYaaaaaaaaaZaaaaaaaaaA.txt
#7 DONE 0.3s
#8 exporting to image
...
Please re-open if this is still happening but couldn't repro from my end.