edannenberg
Build always fails: PermissionError: [Errno 13] Permission denied and alike
OS: Gentoo hardened amd64 Kubler version: 0.9.8
Build isn't always failing at the same stage. Sometime was when trying to download to /distfiles and sometime when writing to /var/tmp/portage. From my last try from scratch the error was: "portage.exception.PermissionDenied: [Errno 13] Permission denied: b'/var/tmp/portage/virtual/libcrypt-1-r1/.ipc/lock'"
I tried deleting everything (kubler clean -N; podman rm -a; podman rmi -a; rm -rf ~/.local/share/containers ~/.kubler) and start from scratch a few times.
I also tried to set BOB_FEATURES="-userfetch -userpriv"
podman info:
host:
arch: amd64
buildahVersion: 1.26.1
cgroupControllers: []
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: app-containers/conmon-2.0.30
path: /usr/libexec/podman/conmon
version: 'conmon version 2.0.30, commit: v2.0.30'
cpuUtilization:
idlePercent: 94.52
systemPercent: 1.32
userPercent: 4.16
cpus: 20
distribution:
distribution: gentoo
version: "2.8"
eventLogger: file
hostname: desktop
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 1000000
size: 1000000000
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 1000000
size: 1000000000
kernel: 5.10.76-gentoo-r1-x86_64
linkmode: dynamic
logDriver: k8s-file
memFree: 2456059904
memTotal: 49300357120
networkBackend: cni
ociRuntime:
name: crun
package: app-containers/crun-1.4.4
path: /usr/bin/crun
version: |-
crun version 1.4.4
commit: 6521fcc5806f20f6187eb933f9f45130c86da230
spec: 1.0.0
+SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: app-containers/slirp4netns-1.2.0
version: |-
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.4
swapFree: 0
swapTotal: 0
uptime: 513h 48m 39.92s (Approximately 21.38 days)
plugins:
log:
- k8s-file
- none
- passthrough
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
docker.io:
Blocked: false
Insecure: false
Location: docker.io
MirrorByDigestOnly: false
Mirrors: null
Prefix: docker.io
PullFromMirror: ""
localhost:5000:
Blocked: false
Insecure: true
Location: localhost:5000
MirrorByDigestOnly: false
Mirrors: null
Prefix: localhost:5000
PullFromMirror: ""
search:
- docker.io
- registry.fedoraproject.org
store:
configFile: /home/andrea/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: vfs
graphOptions: {}
graphRoot: /home/andrea/.local/share/containers/storage
graphRootAllocated: 1992864915456
graphRootUsed: 1618197934080
graphStatus: {}
imageCopyTmpDir: /tmp/.private/andrea
imageStore:
number: 4
runRoot: /run/user/1000/containers
volumePath: /home/andrea/.local/share/containers/storage/volumes
version:
APIVersion: 4.1.0
Built: 1658075888
BuiltTime: Sun Jul 17 18:38:08 2022
GitCommit: e4b03902052294d4f342a185bb54702ed5bed8b1
GoVersion: go1.18.3
Os: linux
OsArch: linux/amd64
Version: 4.1.0
Thanks for the report! Hmm at first glance, this seems to podman related, could you give it a try with Docker to narrow the issue down?
It's also happening with docker, I didn't have it I freshly installed it and I'm using default configuration.
Now I just saw that it tries to build with userpriv usersandbox features enabled. There's another issue talking about that iirc. I tried with BOB_FEATURES and FEATURES in my kubler.conf to disable these features but it's not changing anything
Confirmed. If I use interactive build mode and disable userpriv usersandbox in make.conf it works.
How can I disable them for every image build?
Hmm odd, I'm planning to do the monthly rebuild this Friday, let's see if I can replicate this.
Modifying BOB_FEATURES should be enough to unset userpriv and usersandbox. See man make.conf for all possible options.
I talked to fast. I tried again, I set -userpriv -usersandbox and it fails after doing kubler clean -N ; sudo rm -rf ~/.kubler ~/.local/share/containers ; kubler update && kubler build experiments/minimal with both docker and podman.... I try once again and send logs
docker info:
Client:
Context: default
Debug Mode: false
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 21
Server Version: 20.10.12
Storage Driver: fuse-overlayfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc version: f46b6ba2c9314cfc8caae24a32ec5fe9ef1059fe
init version: de40ad007797e0dcd8b7126f27bb87401d224240
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.76-gentoo-r1-x86_64
Operating System: Gentoo Linux
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 45.91GiB
Name: desktop
ID: 236Q:XUCG:2OPI:OPOI:QEFX:UOCA:5HRC:ANUE:5TMX:JNY2:3SJT:KIQX
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
build.log
»»» jue 28 jul 2022 10:29:24 CEST »»» exec: run_image kubler/bob-musl:20220728 experiments/minimal false rootfs-builder-experiments-minimal-24563-10689
]0;emerge!!! It seems /run is not mounted. Process management may malfunction.
]0;@experiments-minimal:/]0;emerge!!! It seems /run is not mounted. Process management may malfunction.
]0;@experiments-minimal:/]0;emerge!!! It seems /run is not mounted. Process management may malfunction.
[33;01m * IMPORTANT:[39;49;00m 4 news items need reading for repository 'gentoo'.
[33;01m *[39;49;00m Use [32;01meselect news read[39;49;00m to view new items.
[32mThese are the packages that would be merged, in order:[39;49;00m
Calculating dependencies - | / \ \ \ - \ - \ \ / | \ \ | / / - / \ | \ \ | | / \ - - | / \... done!
[[32mebuild[39;49;00m [32;01mN[39;49;00m ] [32msys-libs/musl-1.2.3::gentoo[39;49;00m [32mto /emerge-root/[39;49;00m USE="[34;01m-headers-only[39;49;00m [34;01m-verify-sig[39;49;00m" 1060 KiB
[[32mebuild[39;49;00m [33;01mR[39;49;00m ] [32mvirtual/libcrypt-1-r1:0/1::gentoo[39;49;00m USE="[32;01mstatic-libs[39;49;00m*" 0 KiB
[[32mebuild[39;49;00m [32;01mN[39;49;00m ] [32mvirtual/libcrypt-1-r1:0/1::gentoo[39;49;00m [32mto /emerge-root/[39;49;00m USE="[31;01mstatic-libs[39;49;00m" 0 KiB
[[32;01mebuild[39;49;00m [32;01mN[39;49;00m ] [32;01msys-apps/busybox-1.34.1::gentoo[39;49;00m [32mto /emerge-root/[39;49;00m USE="[31;01mmake-symlinks[39;49;00m [31;01mstatic[39;49;00m [34;01m-debug[39;49;00m [34;01m-ipv6[39;49;00m [34;01m-livecd[39;49;00m [34;01m-math[39;49;00m [34;01m-mdev[39;49;00m [34;01m-pam[39;49;00m [34;01m-savedconfig[39;49;00m ([34;01m-selinux[39;49;00m) [34;01m-sep-usr[39;49;00m [34;01m-syslog[39;49;00m ([34;01m-systemd[39;49;00m)" 2419 KiB
Total: 4 packages (3 new, 1 reinstall), Size of downloads: 3478 KiB
>>> Verifying ebuild manifestsE>>> Jobs: [32m0[39;49;00m of [32m4[39;49;00m complete Load avg: 25.0, 26.8, 25.8]0;experiments-minimal: Jobs: 0 of 4 complete Load avg: 25.0, 26.8, 25.8
[K>>> Jobs: [32m0[39;49;00m of [32m4[39;49;00m complete, [32m1[39;49;00m running Load avg: 25.0, 26.8, 25.8]0;experiments-minimal: Jobs: 0 of 4 complete, 1 running Load avg: 25.0, 26.8, 25.8
[K>>> Emerging ([33;01m1[39;49;00m of [33;01m4[39;49;00m) [32msys-libs/musl-1.2.3::gentoo[39;49;00m for /emerge-root/E>>> Jobs: [32m0[39;49;00m of [32m4[39;49;00m complete, [32m1[39;49;00m running Load avg: 25.0, 26.8, 25.8]0;experiments-minimal: Jobs: 0 of 4 complete, 1 running Load avg: 25.0, 26.8, 25.8
[K>>> [31;01mFailed[39;49;00m to emerge [32msys-libs/musl-1.2.3[39;49;00m for /emerge-root/, Log file:E>>> Jobs: [32m0[39;49;00m of [32m4[39;49;00m complete, [32m1[39;49;00m running Load avg: 25.0, 26.8, 25.8]0;experiments-minimal: Jobs: 0 of 4 complete, 1 running Load avg: 25.0, 26.8, 25.8
[K>>> '[32m/var/tmp/portage/sys-libs/musl-1.2.3/temp/build.log[39;49;00m'E>>> Jobs: [32m0[39;49;00m of [32m4[39;49;00m complete, [32m1[39;49;00m running Load avg: 25.0, 26.8, 25.8]0;experiments-minimal: Jobs: 0 of 4 complete, 1 running Load avg: 25.0, 26.8, 25.8
[K>>> Jobs: [32m0[39;49;00m of [32m4[39;49;00m complete, [32m1[39;49;00m running, [32m1[39;49;00m failed Load avg: 25.0, 26.8, 25.8]0;experiments-minimal: Jobs: 0 of 4 complete, 1 running, 1 failed Load avg: 25.0, 26.8, 25.8
[K>>> Jobs: [32m0[39;49;00m of [32m4[39;49;00m complete, [32m1[39;49;00m failed Load avg: 25.0, 26.8, 25.8]0;experiments-minimal: Jobs: 0 of 4 complete, 1 failed Load avg: 25.0, 26.8, 25.8Ebash: line 1: /distfiles/.__portage_test_write__: Permission denied
!!! No write access to '/distfiles'
!!! No write access to '/distfiles'
!!! File .layout.conf.ftp.snt.utwente.nl isn't fetched but unable to get it.
!!! File musl-1.2.3.tar.gz isn't fetched but unable to get it.
[31;01m * [39;49;00mFetch failed for 'sys-libs/musl-1.2.3', Log file:
[31;01m * [39;49;00m '/var/tmp/portage/sys-libs/musl-1.2.3/temp/build.log'
[32m * [39;49;00mMessages for package [32msys-libs/musl-1.2.3[39;49;00m merged to /emerge-root/:
[31;01m * [39;49;00mFetch failed for 'sys-libs/musl-1.2.3', Log file:
[31;01m * [39;49;00m '/var/tmp/portage/sys-libs/musl-1.2.3/temp/build.log'
]0;@experiments-minimal:/[33m»[[31m✘[33m]»(B[m[33m[(B[mexperiments/minimal[33m]»(B[m fatal: Failed to run image kubler/bob-musl:20220728
Files in kubler's distfiles are owned by root:portage
Did you revert the userpriv and usersandbox changes? My distfiles folder looks like this:
drwxrwxr-x 3 ed portage 132K Jun 28 18:16 distfiles
As it has write permissions for the group, portage has no issue downloading stuff. All files are owned by portage:portage in the folder. Can you double check the write permission for the folder?
portage group has write permission to ~/.kubler/distfiles directory and files inside it. I didn't change features. I even have them twice unset
grep -H userpri /etc/kubler.conf experiments/images/minimal/build.sh /etc/kubler.conf:BOB_FEATURES="${BOB_FEATURES:--parallel-fetch nodoc noinfo noman binpkg-multi-instance -ipc-sandbox -network-sandbox -pid-sandbox -userpriv -usersandbox}"
experiments/images/minimal/build.sh: echo 'FEATURES="-userpriv -usersandbox"' >> /etc/portage/make.conf
Ok, so portage should be running as root but can't write anyways. Do you have some extra hardening on the host that might prevent docker/podman from writing to a host mount?
I'm using a gentoo hardened profile, but afaik i didnt change anything from defaults for security related config. SELinux is disabled, and I don't know what else I could have nor how i could debug it
Hmm let's try to narrow it down:
docker run -it --rm -v /path/to/distfiles:/distfiles busybox
# echo test > /distfiles/foo.txt
If that fails there is most likely some host related issue.
Hmm let's try to narrow it down:
docker run -it --rm -v /path/to/distfiles:/distfiles busybox # echo test > /distfiles/foo.txtIf that fails there is most likely some host related issue.
It's working fine
Ok progess. :)
If I use interactive build mode and disable userpriv usersandbox in make.conf it works.
Let's check how the permissions for /distfiles look from inside the interactive build container.
kubler clean -N ; sudo rm -rf ~/.kubler ~/.local/share/containers ; kubler update && kubler build -i experiments/minimal
kubler-bob-musl / # ls -la /distfiles/
total 174684
drwxrwxr-x+ 1 1000 portage 1052 Jul 28 15:57 .
drwxr-xr-x 24 root root 0 Jul 28 18:46 ..
-rw-rw-r--+ 1 root portage 45 Nov 5 2019 .layout.conf.ftp.snt.utwente.nl
-rw-rw-r--+ 1 root portage 119 Jul 28 07:22 .mirror-cache.json
-rw-rw-r--+ 1 root portage 158456 Mar 8 2017 UnicodeData-10.0.0.txt.xz
-rw-rw-r--+ 1 root portage 311004 Jul 25 2020 bash-completion-2.11.tar.xz
-rw-rw-r--+ 1 root portage 3539 May 25 2019 bashcomp-2.0.3.tar.gz
-rw-rw-r--+ 1 root portage 2105561 May 18 06:52 cython-0.29.30.gh.tar.gz
-rw-rw-r--+ 1 root portage 639864 Jun 4 09:58 eix-0.36.3.tar.xz
-rw-rw-r--+ 1 root portage 8543 Jan 13 2022 eselect-repository-12.tar.gz
-rw-rw-r--+ 1 root portage 16767 May 24 2013 flaggie-0.2.1.tar.bz2
-rw-rw-r--+ 1 root root 5 Jul 28 15:57 foo.txt
-rw-rw-r--+ 1 root portage 21508 Feb 10 2019 gentoo-bashcomp-20190211.tar.bz2
-rw-rw-r--+ 1 root portage 3203805 Mar 2 2021 gentoolkit-0.5.1.tar.gz
-rw-rw-r--+ 1 root portage 6874520 Jan 29 01:46 git-2.35.1.tar.xz
-rw-rw-r--+ 1 root portage 497284 Jan 29 01:46 git-manpages-2.35.1.tar.xz
-rw-rw-r--+ 1 root portage 125758119 Jul 23 2021 go-linux-amd64-bootstrap-1.16.6.tbz
-rw-rw-r--+ 1 root portage 22845866 Jul 12 19:40 go1.18.4.src.tar.gz
-rw-rw-r--+ 1 root portage 1181867 Nov 10 2020 jq-1.7_pre20201109.tar.gz
-rw-rw-r--+ 1 root portage 960663 Jul 2 05:52 lxml-4.9.1.gh.tar.gz
-rw-rw-r--+ 1 root portage 1585293 Nov 17 2010 miscfiles-1.5.tar.gz
-rw-rw-r--+ 1 root portage 944148 Apr 29 04:51 onig-6.9.8.tar.gz
-rw-rw-r--+ 1 root portage 1820282 Feb 23 11:37 openssh-8.9p1.tar.gz
-rw-rw-r--+ 1 root portage 9864061 Jul 5 10:09 openssl-1.1.1q.tar.gz
-rw-rw-r--+ 1 root portage 2839 Sep 7 2020 push-3.4.tar.gz
-rw-rw-r--+ 1 root portage 11128 Aug 9 2020 quoter-4.2.tar.gz
Sorry for the delay, I hope you could resolve the issue, it looked liked something specific to your setup as I couldn't replicate the problem. Feel free to reopen if you still need help with this.