singularity
singularity copied to clipboard
sylabs
Can't pull signed images from registry where /etc/containers/policy.json enforces signature checks
attempting to pull the redhat RHEL 8 universal base image fails when running on RHEL8
[rbello@cuttlefishdev1 rhel-kernel-dev]$ singularity pull docker://registry.access.redhat.com/ubi8/ubi
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Checking if image destination supports signatures
FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: initializing source oci:/home/rbello/.singularity/cache/blob:f20fb774c96377b793475021aca89909da74e9da05136eb6e824aa16f85f22db: Can not copy signatures to oci:/home/rbello/.singularity/cache/blob:f20fb774c96377b793475021aca89909da74e9da05136eb6e824aa16f85f22db: Pushing signatures for OCI images is not supported
[rbello@cuttlefishdev1 rhel-kernel-dev]$ singularity --version
singularity-ce version 3.9.7-1.el8
[rbello@cuttlefishdev1 rhel-kernel-dev]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.5 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.5 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.5"
note other public images seem to work just fine...
for example:
[rbello@cuttlefishdev1 rhel-kernel-dev]$ singularity exec docker://library/ubuntu:18.04 cat /etc/os-release
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 08a6abff8943 done
Copying config 5a9a7d2d1a done
Writing manifest to image destination
Storing signatures
2022/04/07 10:33:53 info unpack layer: sha256:08a6abff89437fab99b52abbefed82ea907f12845c30eeb94f6b93c69be93166
INFO: Creating SIF file...
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Thanks for the information. It looks like the behavior of the containers/image dependency that we use to retrieve and manage OCI layers prior to conversion into Singularity's SIF formats may have changed with regard to Red Hat's implementation of signing. I thought that our use of containers/image was setup to essentially ignore these signatures, but there might be more to do now, or we might never have actually been ignoring policy completely.
I think we pull a UBI image in our tests, but because our CI runs on Ubuntu, there won't be any Red Hat signature config on the system like there is on RHEL 8 so it's very possible we wouldn't see this error. Maybe we pull them from Docker Hub rather than Red Hat's own registry, also.
Arguably perhaps we should expect and fail on the error as the dependency is applying system policy from /etc/containers/policy.json as intended... and since Singularity is using containers/image for OCI image handling we should respect that.
A workaround might be to set "type": "insecureAcceptAnything" in /etc/containers/policy.json for registry.access.redhat.com. However this will stop signature checks for other tools like podman, so is probably not advised.
I'd probably suggest using the Docker Hub mirror of UBI image for now, until we can decide how to tackle this properly given that the system is configured to require signatures for the Red Hat registry: https://hub.docker.com/r/redhat/ubi8 ...
9:47 AM $ singularity pull docker://registry.access.redhat.com/ubi8/ubi
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Checking if image destination supports signatures
FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: initializing source oci:/home/dtrudg/.singularity/cache/blob:f20fb774c96377b793475021aca89909da74e9da05136eb6e824aa16f85f22db: Can not copy signatures to oci:/home/dtrudg/.singularity/cache/blob:f20fb774c96377b793475021aca89909da74e9da05136eb6e824aa16f85f22db: Pushing signatures for OCI images is not supported
09:51 AM $ singularity pull docker://redhat/ubi8
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 4eef1fa1f1c1 done
Copying blob eb24191cef20 done
Copying config 661556e94c done
Writing manifest to image destination
Storing signatures
2022/04/07 09:52:12 info unpack layer: sha256:4eef1fa1f1c17f9ad6a8187dd5a483e11bd340fc116057bd4cece92305151072
2022/04/07 09:52:15 warn rootless{usr/bin/newgidmap} ignoring (usually) harmless EPERM on setxattr "security.capability"
2022/04/07 09:52:15 warn rootless{usr/bin/newuidmap} ignoring (usually) harmless EPERM on setxattr "security.capability"
2022/04/07 09:52:15 info unpack layer: sha256:eb24191cef200934f4fe601f3dc7e7847ea86ce5c52cb9859361bf8e00fe95e7
INFO: Adding owner write permission to build path: /tmp/build-temp-2670086647/rootfs
INFO: Creating SIF file...
thanks for the: The docker ubi image allowed the immediate test to pass, which was just basic "pull"
However, i want to be able to use this as a base repo to provide an environment to build linux kernels.
When i used this as my ".def" file.
Bootstrap: docker
From: redhat/ubi8
%setup
%files
%environment
%post
yum makecache
yum repolist
yum install -y rpm-build \
rpmdevtools \
I got this:
This system is not registered with an entitlement server. You can use subscription-manager to register.
repo id repo name
ubi-8-appstream Red Hat Universal Base Image 8 (RPMs) - AppStream
ubi-8-baseos Red Hat Universal Base Image 8 (RPMs) - BaseOS
ubi-8-codeready-builder Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder
+ yum install -y rpm-build rpmdevtools
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered with an entitlement server. You can use subscription-manager to register.
Last metadata expiration check: 0:00:01 ago on Fri Apr 8 13:22:13 2022.
No match for argument: rpmdevtools
Error: Unable to find a match: rpmdevtools
Oddly, the same first few command fed into a podman build, work and rpmdevtools gets installed.
singularity's view of that rpm:
[rbello@cuttlefishdev1 rhel-kernel-dev]$ singularity run docker://redhat/ubi8 yum info rpmdevtools
INFO: Using cached SIF image
Failed to set locale, defaulting to C.UTF-8
Not root, Subscription Management repositories not updated
This system is not registered with an entitlement server. You can use subscription-manager to register.
Red Hat Universal Base Image 8 (RPMs) - BaseOS 1.8 MB/s | 798 kB 00:00
Red Hat Universal Base Image 8 (RPMs) - AppStream 6.8 MB/s | 2.6 MB 00:00
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 24 kB/s | 17 kB 00:00
Error: No matching Packages to list
and podman's view:
[rbello@cuttlefishdev1 rhel-kernel-dev]$ podman run docker://redhat/ubi8 yum info rpmdevtools
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 MB/s | 39 MB 00:00
Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 45 MB 00:00
Red Hat Universal Base Image 8 (RPMs) - BaseOS 2.1 MB/s | 798 kB 00:00
Red Hat Universal Base Image 8 (RPMs) - AppStre 8.1 MB/s | 2.6 MB 00:00
Red Hat Universal Base Image 8 (RPMs) - CodeRea 91 kB/s | 17 kB 00:00
Available Packages
Name : rpmdevtools
Version : 8.10
Release : 8.el8
Architecture : noarch
Size : 87 k
Source : rpmdevtools-8.10-8.el8.src.rpm
Repository : rhel-8-for-x86_64-appstream-rpms
Summary : RPM Development Tools
URL : https://pagure.io/rpmdevtools
License : GPLv2+ and GPLv2
Description : This package contains scripts and (X)Emacs support files to aid in
: development of RPM packages.
: rpmdev-setuptree Create RPM build tree within user's home directory
: rpmdev-diff Diff contents of two archives
: rpmdev-newspec Creates new .spec from template
: rpmdev-rmdevelrpms Find (and optionally remove) "development" RPMs
: rpmdev-checksig Check package signatures using alternate RPM keyring
: rpminfo Print information about executables and libraries
: rpmdev-md5/sha* Display checksums of all files in an archive file
: rpmdev-vercmp RPM version comparison checker
: spectool Expand and download sources and patches in specfiles
: rpmdev-wipetree Erase all files within dirs created by rpmdev-setuptree
: rpmdev-extract Extract various archives, "tar xvf" style
: rpmdev-bumpspec Bump revision in specfile
: ...and many more.
note also, they produce different repolist's
[rbello@cuttlefishdev1 rhel-kernel-dev]$ singularity run docker://redhat/ubi8 yum repolist
INFO: Using cached SIF image
Failed to set locale, defaulting to C.UTF-8
Not root, Subscription Management repositories not updated
This system is not registered with an entitlement server. You can use subscription-manager to register.
repo id repo name
ubi-8-appstream Red Hat Universal Base Image 8 (RPMs) - AppStream
ubi-8-baseos Red Hat Universal Base Image 8 (RPMs) - BaseOS
ubi-8-codeready-builder Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder
[rbello@cuttlefishdev1 rhel-kernel-dev]$ podman run docker://redhat/ubi8 yum repolist
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
repo id repo name
rhel-8-for-x86_64-appstream-rpms Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
rhel-8-for-x86_64-baseos-rpms Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
ubi-8-appstream Red Hat Universal Base Image 8 (RPMs) - AppStream
ubi-8-baseos Red Hat Universal Base Image 8 (RPMs) - BaseOS
ubi-8-codeready-builder Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder
This system is not registered with an entitlement server. You can use subscription-manager to register.
Oddly, the same first few command fed into a podman build, work and rpmdevtools gets installed.
This is expected behavior I'm afraid. Adding things outside of the minimal UBI package set requires a Red Hat subscription in order to access the standard full repos etc,
podman, as it originates from Red Hat, has some special magic to automatically enable the container to benefit from the host subscription, on a licensed RHEL host. If you do add non-UBI packages, however... you can't generally distribute the resulting container.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/assembly_adding-software-to-a-running-ubi-container_building-running-and-managing-containers
As noted in the link, to restrict yourself to the UBI package set use e.g.
yum install --disablerepo=* --enablerepo=ubi-8-appstream --enablerepo=ubi-8-baseos bzip2
If you need to use licensed repos you will have to manually bind subscription manager files into the container, or use subscription manager in the container.
is it possible to bind during the build step?
Yes - you can use --bind with build with the caveat that the destination must exist in the container rootfs. It will not be created for you.
If it does not exist you can create it with a mkdir in the %setup section. Be very careful in %setup... it runs in the host context.
E.g.
%setup
mkdir $SINGULARITY_ROOTFS/will-bind-here
@ralphbellofatto57 - we'd actually be quite interested in the reasons for pursuing this in Singularity, rather than podman, given your entire stack (host / container / build target) it Red Hat. We're trying to collect more non-HPC batch use cases as we look to SingularityCE 4.0 and beyond, so would much appreciate any info in that regard.
I have been using singularity to build development (compile) environment containers because they just work in the users home directory and bring over the user's id and group membership...
I wanted to encapsulate everything in a container to avoid having to install all the prerequisites into a bare-metal system.
When I ran into the first problems i also attempted to run in podman ran into problems getting three packages required for the kernel development.
specifically these:
yum -y install \
libbpf-devel \
libbabeltrace-devel \
dwarves
so, so far neither environment has been successful but for different reasons...
On Fri, Apr 8, 2022 at 9:58 AM Dave Trudgian @.***> wrote:
@ralphbellofatto57 https://github.com/ralphbellofatto57 - we'd actually be quite interested in the reasons for pursuing this in Singularity, rather than podman, given your entire stack (host / container / build target) it Red Hat. We're trying to collect more non-HPC batch use case as we look to SingularityCE 4.0 and beyond, so would much appreciate any info in that regard.
— Reply to this email directly, view it on GitHub https://github.com/sylabs/singularity/issues/697#issuecomment-1092889173, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIJQGQD5ZZX4CTI5UVNVVE3VEA3Q7ANCNFSM5SZSX3TQ . You are receiving this because you were mentioned.Message ID: @.***>
Portions of this will be tidied up by #2022 - switching to a library for registry interaction that doesn't have handling of podman related config files within it.
This is no longer applicable - we no longer use the container/* series of libraries when pulling OCI containers, so this configuration file is not considered.