buildah
buildah copied to clipboard
containers
Rootless: permission denied adding .tar file
BUG REPORT INFORMATION
Description
This problem occurs only running rootless with buildah unshare.... When running rootless, buildah reports:
error adding content to container "working-container": error extracting "image.in.tar" into "/home/gocdci/.local/share/containers/storage/overlay/11690c1cf9487f46a46bf25417cec5cd690bf415b2c6d573fb21fb3c07027920/merged": Error processing tar file(exit status 1): permission denied
Steps to reproduce the issue:
./mc # run script --- see below for contents
Possible Cause: https://github.com/containers/buildah/blob/5b056ee652bb6165a462305a2ce47745265ba122/troubleshooting.md#5-rootless-buildah-build-fails-eperm-on-nfs
Is this the issue?
Describe the results you received:
error adding content to container "working-container": error extracting image.in.tar into /home/gocdci/.local/share/containers/storage/overlay/11690c1cf9487f46a46bf25417cec5cd690bf415b2c6d573fb21fb3c07027920/merged: Error processing tar file(exit status 1): permission denied
Note that after buildah returns the directory .../11690c1cf9487f46a46bf25417cec5cd690bf415b2c6d573fb21fb3c07027920/... does not exist. Seems removed.
Describe the results you expected:
Storing signatures
ffd216cb14d98916202de79a0d418c53969e1d4c078a85c92c5249e0f8e08595
Output of rpm -q buildah or `apt list buildah:
buildah-1.11.6-12.el7_9.x86_64
Output of `buildah version:
/bin/buildah --version
buildah version 1.11.6 (image-spec 1.0.1-dev, runtime-spec 1.0.1-dev)
I'm gonna guess: stupidly old: try on current code ... which I will do.
Output of podman version if reporting a podman build issue:
NOT-APPLICABLE; no podman used
Output of cat /etc/*release:
# cat /etc/*release*
cat: /etc/lsb-release.d: Is a directory
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.9 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"
Red Hat Enterprise Linux Server release 7.9 (Maipo)
Red Hat Enterprise Linux Server release 7.9 (Maipo)
cpe:/o:redhat:enterprise_linux:7.9:ga:server
Output of uname -a:
Linux dev 3.10.0-1160.15.2.el7.x86_64 #1 SMP Thu Jan 21 16:15:07 EST 2021 x86_64 x86_64 x86_64 GNU/Linux
Output of cat /etc/containers/storage.conf:
# cat cat /etc/containers/storage.conf
cat: cat: No such file or directory
# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]
# Default Storage Driver
driver = "overlay"
# Temporary storage location
runroot = "/var/run/containers/storage"
# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"
[storage.options]
# Storage options to be passed to underlying storage drivers
# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]
# Size is used to set a maximum size of the container image. Only supported by
# certain container storage drivers.
size = ""
# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"
# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs. Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536
# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file. Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"
[storage.options.thinpool]
# Storage Options for thinpool
# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"
# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"
# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"
# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"
# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper
# directlvm_device = ""
# directlvm_device_force wipes device even if device already has a filesystem
# directlvm_device_force = "True"
# fs specifies the filesystem type to use for the base device.
# fs="xfs"
# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"
# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"
# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""
# mountopt specifies extra mount options used when mounting the thin devices.
# mountopt = ""
# use_deferred_removal Marking device for deferred removal
# use_deferred_removal = "True"
# use_deferred_deletion Marking device for deferred deletion
# use_deferred_deletion = "True"
# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"
MC helper script
#!/bin/bash
VERS="1.41"
NAME="tsaimrec-dbw"
TAR_IN="${PWD}/image.in.tar"
rm -rf ./containers ~/.containers ~/.local ./overlay*
export XDG_RUNTIME_DIR=$PWD
HELPER="./buildah_helper.sh"
rm -f $HELPER
cat > $HELPER << EOF
#!/bin/sh
# Create a container
echo make scratch container
CONTAINER=\$(buildah from scratch)
# Mount the container filesystem
echo mount container
MOUNTPOINT=\$(buildah mount \$CONTAINER)
buildah add \$CONTAINER './image.in.tar' '/'
echo add apps
buildah add \$CONTAINER 'my.tsk' '/bin/my.tsk'
echo add author
buildah config --author "gocdci ([email protected])" \$CONTAINER
# Cleanup
echo unmount
buildah unmount \$CONTAINER
# Save the container to an image
echo commit
buildah commit --squash --rm working-container ${NAME}:${VERS}
echo login
buildah login -u gocdci --password-stdin < ./auth.json
echo push
buildah push "localhost/${NAME}:${VERS}" "<myurl>/${NAME}:${VERS}"
EOF
chmod 500 $HELPER
echo running ${HELPER} rootless ...
buildah unshare ${HELPER} --log-level=debug
echo done
# buildah push "localhost/${NAME}:${VERS}" "docker-daemon:ts/${NAME}:${VERS}"
buildah images
buildah containers
exit 0
Possibly Related: Weird uid/gid
Upon looking in the files buildah makes there are some files with what appears to be a bad uid/guid. These numbers do not correspond to anything I can find anywhere on my box or inside the tar file which has its own /etc/passwd: Note 18313466:
$ find /home/gocdci/.local/share/containers -ls | grep 18313466
22300588 0 drwxr-xr-x 2 18313466 18313466 114 Oct 1 17:49 /home/gocdci/.local/share/containers/storage/overlay/5a1c320e4160aa1c3d4b83908aade8c15dd4cd0eacf45faee8ef8de4c17f9035/diff/bin
22300589 1008 -rwxr-xr-x 1 18313466 18313466 1029624 Nov 12 2014 /home/gocdci/.local/share/containers/storage/overlay/5a1c320e4160aa1c3d4b83908aade8c15dd4cd0eacf45faee8ef8de4c17f9035/diff/bin/bash
22300590 52 -rwxr-xr-x 1 18313466 18313466 51912 Mar 14 2015 /home/gocdci/.local/share/containers/storage/overlay/5a1c320e4160aa1c3d4b83908aade8c15dd4cd0eacf45faee8ef8de4c17f9035/diff/bin/cat
22300591 16 -rwxr-xr-x 1 18313466 18313466 14560 Sep 8 2014 /home/gocdci/.local/share/containers/storage/overlay/5a1c320e4160aa1c3d4b83908aade8c15dd4cd0eacf45faee8ef8de4c17f9035/diff/bin/chacl
22300592 56 -rwxr-xr-x 1 18313466 18313466 55944 Mar 14 2015 /home/gocdci/.local/share/containers/storage/overlay/5a1c320e4160aa1c3d4b83908aade8c15dd4cd0eacf45faee8ef8de4c17f9035/diff/bin/chmod
22300593 64 -rwxr-xr-x 1 18313466 18313466 64168 Mar 14 2015 /home/gocdci/.local/share/containers/storage/overlay/5a1c320e4160aa1c3d4b83908aade8c15dd4cd0eacf45faee8ef8de4c17f9035/diff/bin/chown
22300594 64 -rwxr-xr-x 1 18313466 18313466 64168 Mar 14 2015 /home/gocdci/.local/share/containers/storage/overlay/5a1c320e4160aa1c3d4b83908aade8c15dd4cd0eacf45faee8ef8de4c17f9035/diff/bin/date
22300595 52 -rwxr-xr-x 1 18313466 18313466 52224 Mar 29 2015 /home/gocdci/.local/share/containers/storage/overlay/5a1c320e4160aa1c3d4b83908aade8c15dd4cd0eacf45faee8ef8de4c17f9035/diff/bin/dmesg
22300596 0 lrwxrwxrwx 1 18313466 18313466 8 Oct 1 17:49 /home/gocdci/.local/share/containers/storage/overlay/5a1c320e4160aa1c3d4b83908aade8c15dd4cd0eacf45faee8ef8de4c17f9035/diff/bin/dnsdomainname -> hostname
An error extracting an archive part of the way through suggests that we hit a permissions error trying to create something we encountered part of the way through the archive. If you can share a listing of the archive's contents ("tar tvf image.in.tar"), that will help us figure out which item came after "bin/dnsdomainname" in the archive. If the error was encountered while attempting to restore extended attributes, they won't be shown, but it'll help us figure out what's going on.
As to the cause, is your home directory on NFS, as you suspect? The filesystem type will be provided as the "Type:" in the output of stat -f $HOME.
stat -f $HOME
File: "/home/gocdci"
ID: fd0100000000 Namelen: 255 Type: xfs
Block size: 4096 Fundamental block size: 4096
Blocks: Total: 5240059 Free: 2912969 Available: 2912969
Inodes: Total: 10485232 Free: 10114610
tar file snippet. Please note extraction and image building under buildah is done also on user gocdci. Now, in reality these files should be owned by root, however because of this error, I tried changing the user to gocdci. Since either owner in the tar file leads to the same error, I believe the issue does not depend on the owner of the file in the .tar. Maybe the issue here is dealing with links confusing tar contents with /bin/dnsdomainname on the host file system?
tar tvf image.in.tar ...
.
.
.
drwxr-xr-x gocdci/gocdci 0 2021-08-17 19:57 bin/
-rwxr-xr-x gocdci/gocdci 1029624 2014-11-12 18:08 bin/bash
-rwxr-xr-x gocdci/gocdci 51912 2015-03-14 11:47 bin/cat
-rwxr-xr-x gocdci/gocdci 14560 2014-09-08 03:01 bin/chacl
-rwxr-xr-x gocdci/gocdci 55944 2015-03-14 11:47 bin/chmod
-rwxr-xr-x gocdci/gocdci 64168 2015-03-14 11:47 bin/chown
-rwxr-xr-x gocdci/gocdci 64168 2015-03-14 11:47 bin/date
-rwxr-xr-x gocdci/gocdci 52224 2015-03-29 18:34 bin/dmesg
lrwxrwxrwx gocdci/gocdci 0 2013-11-03 09:41 bin/dnsdomainname -> hostname
lrwxrwxrwx gocdci/gocdci 0 2013-11-03 09:41 bin/domainname -> hostname
-rwxr-xr-x gocdci/gocdci 31208 2015-03-14 11:47 bin/echo
-rwxr-xr-x gocdci/gocdci 29 2015-02-13 20:27 bin/egrep
-rwxr-xr-x gocdci/gocdci 27080 2015-03-14 11:47 bin/false
-rwxr-xr-x gocdci/gocdci 29 2015-02-13 20:27 bin/fgrep
-rwxr-xr-x gocdci/gocdci 23584 2014-09-08 03:01 bin/getfacl
-rwxr-xr-x gocdci/gocdci 202936 2015-02-13 20:27 bin/grep
-rwxr-xr-x gocdci/gocdci 2301 2014-09-26 15:41 bin/gunzip
-rwxr-xr-x gocdci/gocdci 5927 2014-09-26 15:41 bin/gzexe
-rwxr-xr-x gocdci/gocdci 14640 2013-11-03 09:41 bin/hostname
-rwxr-xr-x gocdci/gocdci 314560 2014-09-05 10:57 bin/ip
-rwxr-xr-x gocdci/gocdci 22960 2015-03-06 16:13 bin/kill
-rwxr-xr-x gocdci/gocdci 118280 2015-03-14 11:47 bin/ls
-rwxr-xr-x gocdci/gocdci 80744 2015-03-14 11:47 bin/mkdir
-rwxr-xr-x gocdci/gocdci 39528 2015-03-14 11:47 bin/mktemp
lrwxrwxrwx gocdci/gocdci 0 2013-11-03 09:41 bin/nisdomainname -> hostname
-rwxr-xr-x gocdci/gocdci 70576 2014-10-28 03:18 bin/ping
.
.
.
Well, you're on an XFS filesystem, which is widely used for this, so it's not related to NFS. 18313466 is likely the ID that your own UID (as given in the archive) is being mapped to in the range that's available for use in the user namespace where your build is running - the range (or ranges) that is available to you is configured in /etc/subuid and /etc/subgid.
The order of entries in the archive listing suggests that the permissions problem was triggered while extracting the domainname symbolic link, but given that extracting the dnsdomainname symbolic link that immediately precedes it succeeded, that doesn't really clear things up for me. A version compiled with a later version of the compiler (buildah-1.11.6-12.el7_9 was built with Go 1.12) would hopefully include the name of the syscall that returned the error, along with the name of the file that it was attempting to create or write, in the error message.
A friendly reminder that this issue had no activity for 30 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
