buildkit icon indicating copy to clipboard operation
buildkit copied to clipboard
verified publisher icon moby

[v0.13 regression] buildkit/runc-native/snapshots/snapshots/10/var/cache/apt/archives/partial: permission denied

Open reimgun opened this issue 1 year ago • 5 comments

buildkit version tried: 0.13.0 and 0.13.1

Error Message:

#9 [internal] load build context
#9 transferring context: 179B 0.0s done
#9 ERROR: error from sender: open /home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat/.local/share/buildkit/runc-native/snapshots/snapshots/10/var/cache/apt/archives/partial: permission denied
------
 > [internal] load build context:
------
error: failed to solve: error from sender: open /home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat/.local/share/buildkit/runc-native/snapshots/snapshots/10/var/cache/apt/archives/partial: permission denied

with Version 0.12.5 all is working perfectly fine.

reimgun avatar Mar 21 '24 08:03 reimgun

Please provide host OS, kernel version, filesystem, and a (minimal and yet self-contained) reproducer

AkihiroSuda avatar Mar 21 '24 09:03 AkihiroSuda

@AkihiroSuda container information where buildkit is installed: OS: NAME="Red Hat Enterprise Linux" VERSION="9.3 (Plow)" bash-5.1$ uname -a Linux jenkins-6d54b58dc5-2lxtg-gh-ubi-mc3kx 5.10.205-195.807.amzn2.x86_64 #1 SMP Tue Jan 16 18:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

container is running in kubernetes Version 1.27

we run buildkit now more then 1,5 years in that setup.

always worked well until version 0.13.0 and 0.13.1

reimgun avatar Mar 21 '24 12:03 reimgun

Please provide a reproducer

AkihiroSuda avatar Mar 22 '24 16:03 AkihiroSuda

buildctl-daemonless.sh build --frontend dockerfile.v0 --local context=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --local dockerfile=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --opt filename=Dockerfile --metadata-file metadata.json --output 'type=image,"name=****.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:8.12.2.3,*****.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:latest",push=true'

reimgun avatar Apr 08 '24 06:04 reimgun

@AkihiroSuda any update ? i have tested it also with 0.14.0 - it's the same ....

it happens also for other images:

#6 [internal] load build context
#6 transferring context: 129B 0.8s done
#6 ERROR: error from sender: open /home/jenkins/workspace/prod-images-0/at_ubi8_dotnet_radarlive_settingsmanager/build_images/.tar-scan: permission denied
------
 > [internal] load build context:
------
error: failed to solve: error from sender: open /home/jenkins/workspace/prod-images-0/at_ubi8_dotnet_radarlive_settingsmanager/build_images/.tar-scan: permission denied

could it be that it hast problems with .dirs like .local or .tar-scan ?

reimgun avatar Jun 19 '24 11:06 reimgun

@AkihiroSuda

here also the debug logs:

error: failed to solve: error from sender: open /home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat/.local/share/buildkit/runc-overlayfs/snapshots/snapshots/10/work/work: permission denied
7148 v0.14.1 buildkitd --oci-worker-no-process-sandbox --addr=unix:///run/user/1000/buildkit/buildkitd.sock
github.com/tonistiigi/fsutil.(*receiver).run.func2
	/src/vendor/github.com/tonistiigi/fsutil/receive.go:211
golang.org/x/sync/errgroup.(*Group).Go.func1
	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

7148 v0.14.1 buildkitd --oci-worker-no-process-sandbox --addr=unix:///run/user/1000/buildkit/buildkitd.sock
github.com/moby/buildkit/session/filesync.recvDiffCopy
	/src/session/filesync/diffcopy.go:108
github.com/moby/buildkit/session/filesync.FSSync
	/src/session/filesync/filesync.go:235
github.com/moby/buildkit/source/local.(*localSourceHandler).snapshot
	/src/source/local/source.go:264
github.com/moby/buildkit/source/local.(*localSourceHandler).Snapshot
	/src/source/local/source.go:153
github.com/moby/buildkit/solver/llbsolver/ops.(*SourceOp).Exec
	/src/solver/llbsolver/ops/source.go:108
github.com/moby/buildkit/solver.(*sharedOp).Exec.func2
	/src/solver/jobs.go:1100
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
	/src/util/flightcontrol/flightcontrol.go:122
sync.(*Once).doSlow
	/usr/local/go/src/sync/once.go:74
sync.(*Once).Do
	/usr/local/go/src/sync/once.go:65
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

7148 v0.14.1 buildkitd --oci-worker-no-process-sandbox --addr=unix:///run/user/1000/buildkit/buildkitd.sock
github.com/moby/buildkit/solver.(*edge).execOp
	/src/solver/edge.go:979
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
	/src/solver/internal/pipe/pipe.go:82
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

7192 v0.14.1 buildctl --addr=unix:///run/user/1000/buildkit/buildkitd.sock --debug build --frontend dockerfile.v0 --local context=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --local dockerfile=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --opt filename=Dockerfile --metadata-file metadata.json --output type=image,"name=itmpproductshared.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:8.14.2.2,itmpproductshared.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:latest",push=true --opt build-arg:HTTP_PROXY=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:http_proxy=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:HTTPS_PROXY=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:https_proxy=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:NO_PROXY=localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,44.0.0.0/8,.allianz,.aeat.allianz.at,.svc,.cluster.local,jmp.allianz.net,cmp.allianz.net,github.developer.allianz.io --opt build-arg:no_proxy=localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,44.0.0.0/8,.allianz,.aeat.allianz.at,.svc,.cluster.local,jmp.allianz.net,cmp.allianz.net,github.developer.allianz.io --opt build-arg:PROXY_HOST=proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz --opt build-arg:PROXY_PORT=8080 --opt build-arg:NONPROXYHOSTS=localhost|127.*|10.*|172.16.0.0/12|192.168.*|44.*|*.allianz|*.aeat.allianz.at|*.svc|*.cluster.local|jmp.allianz.net|cmp.allianz.net|github.developer.allianz.io
google.golang.org/grpc.(*ClientConn).Invoke
	/src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
	/src/api/services/control/control.pb.go:2234
github.com/moby/buildkit/client.(*Client).solve.func2
	/src/client/solve.go:274
golang.org/x/sync/errgroup.(*Group).Go.func1
	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

7192 v0.14.1 buildctl --addr=unix:///run/user/1000/buildkit/buildkitd.sock --debug build --frontend dockerfile.v0 --local context=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --local dockerfile=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --opt filename=Dockerfile --metadata-file metadata.json --output type=image,"name=itmpproductshared.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:8.14.2.2,itmpproductshared.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:latest",push=true --opt build-arg:HTTP_PROXY=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:http_proxy=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:HTTPS_PROXY=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:https_proxy=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:NO_PROXY=localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,44.0.0.0/8,.allianz,.aeat.allianz.at,.svc,.cluster.local,jmp.allianz.net,cmp.allianz.net,github.developer.allianz.io --opt build-arg:no_proxy=localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,44.0.0.0/8,.allianz,.aeat.allianz.at,.svc,.cluster.local,jmp.allianz.net,cmp.allianz.net,github.developer.allianz.io --opt build-arg:PROXY_HOST=proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz --opt build-arg:PROXY_PORT=8080 --opt build-arg:NONPROXYHOSTS=localhost|127.*|10.*|172.16.0.0/12|192.168.*|44.*|*.allianz|*.aeat.allianz.at|*.svc|*.cluster.local|jmp.allianz.net|cmp.allianz.net|github.developer.allianz.io
github.com/moby/buildkit/client.(*Client).solve.func2
	/src/client/solve.go:290
golang.org/x/sync/errgroup.(*Group).Go.func1
	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

reimgun avatar Jul 04 '24 11:07 reimgun