buildah fails with FROM ghcr.io/distroless/alpine-base:latest

[vdemeester]

Description

buildah doesn't seem to be able to build an image using the ghcr.io/distroless/alpine-base:latest image. I am not entirely sure what is the specificity of this particular image except it's build with apko.

Steps to reproduce the issue:

1. oc run -i --tty meh --image=registry.redhat.io/rhel8/buildah@sha256:0a86ecbdfbe86e9d225b7fe4b090a0dd6d323f8afdfdf2bd933ff223ddb53320 --restart=Never -- sh
2. create a Dockerfile with the following content

   FROM ghcr.io/distroless/alpine-base:latest
   

3. Run `buildah bud --storage-driver=vfs

Describe the results you received:

Build fail with "operator not permitted". This runs as root in a container, in OpenShift.

STEP 1/1: FROM ghcr.io/distroless/alpine-base:latest
Trying to pull ghcr.io/distroless/alpine-base:latest...
Getting image source signatures
Copying blob afb725cdfbe4 done  
Copying blob afb725cdfbe4 done  
error creating build container: copying system image from manifest list: writing blob: adding layer with blob "sha256:afb725cdfbe4a41f1de8dbb3ac124dee12eaabec5398eecd92ee585cd946f018": ApplyLayer stdout:  stderr: operation not permitted exit status 1

Describe the results you expected:

Build succeeds. Note: using the standard alpine image works seamlessly, and docker build on this image works seamlessly as well.

Output of rpm -q buildah or apt list buildah:

(paste your output here)

Output of buildah version:

Version:         1.26.4
Go Version:      go1.18.4
Image Spec:      1.0.2-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.22.0
Git Commit:      
Built:           Mon Aug  8 14:11:10 2022
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of uname -a:

Linux meh 4.18.0-305.57.1.el8_4.x86_64 #1 SMP Tue Jul 5 17:55:55 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# This file is the configuration file for all tools                                                 
# that use the containers/storage library. The storage.conf file                                    
# overrides all other storage.conf files. Container engines using the                               
# container/storage library do not inherit fields from other storage.conf                           
# files.                                                                                            
#                                                                                                   
#  Note: The storage.conf file overrides other storage.conf files based on this precedence:
#      /usr/containers/storage.conf                                                                 
#      /etc/containers/storage.conf                                                                 
#      $HOME/.config/containers/storage.conf                                                        
#      $XDG_CONFIG_HOME/containers/storage.conf (If XDG_CONFIG_HOME is set)     
# See man 5 containers-storage.conf for more information                                            
# The "container storage" table contains all of the server options.                                 
[storage]                                                                                           
                                                  
# Default Storage Driver, Must be set for proper operation.                                         
driver = "overlay"                                                                                  
                                                                                                    
# Temporary storage location                                                                        
runroot = "/run/containers/storage"                                                                 
                                                  
# Primary Read/Write location of container storage                                                  
# When changing the graphroot location on an SELINUX system, you must                               
# ensure  the labeling matches the default locations labels with the                                
# following commands:                                                                               
# semanage fcontext -a -e /var/lib/containers/storage /NEWSTORAGEPATH                               
# restorecon -R -v /NEWSTORAGEPATH                                                                  
graphroot = "/var/lib/containers/storage"   

# Storage path for rootless users                                                                                                                                                              [152/1817]
#                                                                                                   
# rootless_storage_path = "$HOME/.local/share/containers/storage"                                   
                                                                                                    
[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
"/var/lib/shared",
]

# Allows specification of how storage is populated when pulling images. This
# option can speed the pulling process of images compressed with format
# zstd:chunked. Containers/storage looks for files within images that are being
# pulled from a container registry that were previously pulled to the host.  It
# can copy or create a hard link to the existing file when it finds them,
# eliminating the need to pull them from the container registry. These options
# can deduplicate pulling of content, disk storage of content and can allow the
# kernel to use less memory when running containers.

# containers/storage supports four keys
#   * enable_partial_images="true" | "false"
#     Tells containers/storage to look for files previously pulled in storage
#     rather then always pulling them from the container registry.
#   * use_hard_links = "false" | "true"
#     Tells containers/storage to use hard links rather then create new files in
#     the image, if an identical file already existed in storage.
#   * ostree_repos = ""
#     Tells containers/storage where an ostree repository exists that might have
#     previously pulled content which can be used when attempting to avoid
#     pulling content from the container registry
pull_options = {enable_partial_images = "false", use_hard_links = "false", ostree_repos=""}

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to the UIDs/GIDs as they should appear outside of the container,
# and the length of the range of UIDs/GIDs.  Additional mapped sets can be
# listed and will be needed by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and then a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped in-container ID,
# until all of the entries have been used for maps.
#
# remap-user = "containers"
# remap-group = "containers"

# Root-auto-userns-user is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid and /etc/subgid file.  These ranges will be partitioned
# to containers configured to create automatically a user namespace.  Containers
# configured to automatically create a user namespace can still overlap with containers
# having an explicit mapping set.
# This setting is ignored when running as rootless.
# root-auto-userns-user = "storage"
#
# Auto-userns-min-size is the minimum size for a user namespace created automatically.
# auto-userns-min-size=1024
#
# Auto-userns-max-size is the minimum size for a user namespace created automatically.
# auto-userns-max-size=65536

[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids.  Note multiple UIDs will be
# squashed down to the default uid in the container.  These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
#ignore_chown_errors = "false"

# Inodes is used to set a maximum inodes of the container image.
# inodes = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
mount_program = "/usr/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,fsync=0"

# Set to skip a PRIVATE bind mount on the storage home directory.
# skip_mount_home = "false"

# Size is used to set a maximum size of the container image.             
# size = ""                                
                                                  
# ForceMask specifies the permissions mask that is used for new files and                           
# directories.                                                                                      
#
# The values "shared" and "private" are accepted.
# Octal permission masks are also accepted.
#
#  "": No value specified.
#     All files/directories, get set with the permissions identified within the
#     image.
#  "private": it is equivalent to 0700.
#     All files/directories get set with 0700 permissions.  The owner has rwx
#     access to the files. No other users on the system can access the files.
#     This setting could be used with networked based homedirs.
#  "shared": it is equivalent to 0755.
#     The owner has rwx access to the files and everyone else can read, access
#     and execute them. This setting is useful for sharing containers storage
#     with other users.  For instance have a storage owned by root but shared
#     to rootless users as an additional store.
#     NOTE:  All files within the image are made readable and executable by any
#     user on the system. Even /etc/shadow within your image is now readable by
#     any user.
#
#   OCTAL: Users can experiment with other OCTAL Permissions.
#
#  Note: The force_mask Flag is an experimental feature, it could change in the
#  future.  When "force_mask" is set the original permission mask is stored in
#  the "user.containers.override_stat" xattr and the "mount_program" option must
#  be specified. Mount programs like "/usr/bin/fuse-overlayfs" present the
#  extended attribute permissions to processes within containers rather than the
#  "force_mask"  permissions.
#
# force_mask = ""

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# metadata_size is used to set the `pvcreate --metadatasize` options when
# creating thin devices. Default is 128k
# metadata_size = ""

# Size is used to set a maximum size of the container image.
# size = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

[vdemeester]

apko issue : https://github.com/chainguard-dev/apko/issues/344

[flouthoc]

Hi @vdemeester ,

Strange it works for me, Could you try adding --isolation chroot and after this --cap-add=SYS_ADMIN --security-opt seccomp=unconfined --isolation chroot ?

[vdemeester]

@flouthoc didn't work either. I wonder if it's related to https://github.com/tektoncd/pipeline/issues/5188 (and https://github.com/google/ko/pull/776/files or something in apko or github.com/google/go-containerregistry)

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc @vdemeester Still an issue?

[flouthoc]

I am assuming that the issue is resolved and refereed PR/Issues have more context so closing this issue.