gramineproject
gsc build fails for redhat distro
we are calling "gsc build " using configuration "Distro:auto" (since "Distro:redat/ubi9.4" failed) and gsc fails with the error:
Downloading metadata... error: cannot update repo 'rhel-9-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried; Last error: Curl error (60): SSL peer certificate or SSH remote key was not OK for https://cdn.redhat.com/customer/Library/content/dist/rhel9/9/x86_64/baseos/os/repodata/repomd.xml [SSL certificate problem: self-signed certificate in certificate chain]
(here is full log
Building unsigned graminized Docker image gsc-idbroker-nevisauth-unsigned from original application image idbroker-nevisauth...
Warning: Duplicate key loader.env.PATH. Concatenating values from <merged ./manifest.properties and redhat/ubi-minimal/entrypoint.manifest.template> and <idbroker-nevisauth image env>.
Step 1/30 : FROM redhat/ubi9-minimal:9.6 AS gramine
---> e43c1a24fb37 Step 2/30 : COPY redhat.repo /etc/yum.repos.d/
---> ea5155221acc Step 3/30 : COPY pki/entitlement/ /etc/pki/entitlement/
---> c5a9e2467396 Step 4/30 : COPY redhat-uep.pem /etc/rhsm/ca/
---> afdea01efec8 Step 5/30 : RUN rm -rf /etc/rhsm-host && microdnf install -y subscription-manager && rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms && microdnf install -y autoconf bison cmake elfutils-libelf-devel flex gawk gcc-c++ git httpd kernel-headers libevent-devel make nasm ncurses-devel ninja-build openssl-devel patch pkg-config protobuf-c-compiler protobuf-c-devel protobuf-compiler protobuf-devel python3 python3-cryptography python3-pip python3-protobuf python3-voluptuous rpm-build && /usr/bin/python3 -B -m pip install 'tomli>=1.1.0' 'tomli-w>=0.4.0' 'meson>=0.56,!=1.2.*'
---> Running in 7f4c9e7b66ff Downloading metadata... Downloading metadata... Downloading metadata... Downloading metadata... error: cannot update repo 'rhel-9-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried; Last error: Curl error (60): SSL peer certificate or SSH remote key was not OK for https://cdn.redhat.com/customer/Library/content/dist/rhel9/9/x86_64/baseos/os/repodata/repomd.xml [SSL certificate problem: self-signed certificate in certificate chain]
Failed to build unsigned graminized Docker image gsc-idbroker-nevisauth-unsigned.
)
Can you please help us ? Many thanks in advance !
Uwe Albert
@DukeDavis12 Hi you solved our previous issue using PR 238 . Can you please help us with this red hat issue as well ? Many thanks in advance ! Best regards Uwe
when using "Distro:redhat/ubi9:9.4" we get the error:
Building unsigned graminized Docker image gsc-idbroker-nevisauth-unsigned from original application image idbroker-nevisauth...
Warning: Duplicate key loader.env.PATH. Concatenating values from <merged ./manifest.properties and redhat/ubi/entrypoint.manifest.template> and <idbroker-nevisauth image env>.
Step 1/37 : FROM redhat/ubi9:9.4 AS gramine
---> 769453be7412 Step 2/37 : COPY redhat.repo /etc/yum.repos.d/
---> 6c46b0242943 Step 3/37 : COPY pki/entitlement/ /etc/pki/entitlement/
---> 2ebb636689e3 Step 4/37 : COPY redhat-uep.pem /etc/rhsm/ca/
---> 7ff35ec4d83c Step 5/37 : RUN rm -rf /etc/rhsm-host && subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms && dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && dnf update -y && dnf install -y autoconf bison cmake elfutils-libelf-devel flex gawk gcc-c++ git httpd kernel-headers libevent-devel make nasm ncurses-devel ninja-build openssl-devel patch pkg-config protobuf-c-compiler protobuf-c-devel protobuf-compiler protobuf-devel python3 python3-cryptography python3-pip python3-protobuf python3-voluptuous rpm-build && /usr/bin/python3 -B -m pip install 'tomli>=1.1.0' 'tomli-w>=0.4.0' 'meson>=0.56,!=1.2.*'
---> Running in f8295af8e8d0 System certificates corrupted. Please reregister.
@uwe-albert-ibm GSC requires host RHEL system to be subscribed via subscription-manager. In your logs as you can see System certificates corrupted. Please reregister., Can you check registration on your system and try to register it
@anjalirai-intel : Hi state is as follows:
- host system was already registered
- when calling "subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms" locally it works
- whereas within Dockerfile "subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms" fails with error above (... Please reregister)
Sorry for the fuss, but I am confused . What is your recommendation please ?
@anjalirai-intel : Hi state is as follows:
- host system was already registered
- when calling "subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms" locally it works
- whereas within Dockerfile "subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms" fails with error above (... Please reregister)
Sorry for the fuss, but I am confused . What is your recommendation please ?
Can you share your host system os details
@anjalirai-intel sure , here is one piece [root@sekidp-runner03 ubi]# cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="9.4 (Plow)" ID="rhel" ID_LIKE="fedora" VERSION_ID="9.4" PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux 9.4 (Plow)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9" BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.4 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
here is another uname -a Linux sekidp-runner03 5.14.0-427.66.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Apr 20 15:36:45 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux
@uwe-albert-ibm Can you try to re-register once again
I dont know cause it is a managed environment , I will try and let you know
@anjalirai-intel : Hi state is as follows:
- host system was already registered
- when calling "subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms" locally it works
- whereas within Dockerfile "subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms" fails with error above (... Please reregister)
Sorry for the fuss, but I am confused . What is your recommendation please ?
Can you share your host system os details
@anjalirai-intel sure , here is one piece [root@sekidp-runner03 ubi]# cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="9.4 (Plow)" ID="rhel" ID_LIKE="fedora" VERSION_ID="9.4" PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux 9.4 (Plow)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9" BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.4 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
here is another uname -a Linux sekidp-runner03 5.14.0-427.66.1.el9_4.x86_64 https://github.com/gramineproject/gsc/pull/1 SMP PREEMPT_DYNAMIC Sun Apr 20 15:36:45 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux
@uwe-albert-ibm It is probably a RHEL System issue, you can try suggestion mentioned in this url System registration fails with error System certificates corrupted. Please reregister on content host registering to Red Hat Satellite 6
@uwe-albert-ibm It is probably a RHEL System issue, you can try suggestion mentioned in this url System registration fails with error System certificates corrupted. Please reregister on content host registering to Red Hat Satellite 6
@anjalirai-intel many thanks for investigating. I ll try and report
@anjalirai-intel : Hi , we checked for re-registration and figured out that it is using an IBM managed subscription as it is a VSI running on ibm cloud and we cant reregister. May be you have any other idea for solving our issue ?
@anjalirai-intel Hi , I have executed a re-registration on the RHEL host machine which reported successful, but the error remains. Two question sfrom my end please: Since when does GSC support red-hat ? And would it be a mitigation to use distro "centos" instead of red-hat/ubi ? Many thanks in advance !
@anjalirai-intel Hi , I have executed a re-registration on the RHEL host machine which reported successful, but the error remains. Two question sfrom my end please: Since when does GSC support red-hat ? And would it be a mitigation to use distro "centos" instead of red-hat/ubi ? Many thanks in advance !
@uwe-albert-ibm It's been 2 years since GSC support RHEL.
Adding @sahason @DukeDavis12 for better clarification
@DukeDavis12 Hi you solved our previous issue using PR 238 . Can you please help us with this red hat issue as well ? Many thanks in advance ! Best regards Uwe
@uwe-albert-ibm Can you the share link to your previous issue ?
@DukeDavis12 Hi you solved our previous issue using PR 238 . Can you please help us with this red hat issue as well ? Many thanks in advance ! Best regards Uwe
@uwe-albert-ibm Can you the share link to your previous issue ?
@DukeDavis12 I dont have the link, but it was solved by PR 238 https://github.com/gramineproject/gsc/pull/238
@anjalirai-intel Hi , I have executed a re-registration on the RHEL host machine which reported successful, but the error remains. Two question sfrom my end please: Since when does GSC support red-hat ? And would it be a mitigation to use distro "centos" instead of red-hat/ubi ? Many thanks in advance !
@uwe-albert-ibm It's been 2 years since GSC support RHEL.
Adding @sahason @DukeDavis12 for better clarification
@anjalirai-intel Thanks for sharing ! Our IBM Cloud admins confirmed that the machine on which our gsc build runs is definitely registered. I guess we are stuck here, hence I am asking again for your opinion using "centos" as fallback ? (Other components which use "ubuntu 22.04) work on the very same machine) Many thanks in advance for your support !
@uwe-albert-ibm
We currently support only Red Hat-managed subscriptions. As part of this support, when a host is registered, certain entitlement files are copied into the Docker image to enable package installation. Specifically, the following files are copied:
/etc/yum.repos.d/redhat.repo
/etc/pki/entitlement/*
/etc/rhsm/ca/redhat-uep.pem
However, since you're using an IBM-managed Red Hat subscription, additional or different files may be required to successfully install packages inside the Docker image.
To support IBM-managed subscriptions, please identify the full set of required files from the host system that need to be copied into the image. A good starting point would be to include the following files (Please try with these files first):
/etc/rhsm/rhsm.conf
/etc/rhsm/ca/redhat-entitlement-authority.pem
/etc/pki/consumer/
There may be other necessary files depending on the specifics of the IBM-managed setup.
Once the required files are determined, you will need to modify the following parts of the GSC project to copy them into the Docker image:
gsc.py – to include the additional files during the GSC build process
Dockerfile.build.template – to copy the entitlement files during the build stage
Dockerfile.compile.template – to ensure the same during the compile stage
@sahason Many thanks for analysis. I ll try out your solution approach and come back to you.
@uwe-albert-ibm A CentOS base image is also compatible with RHEL host. If you're fine running your workload on CentOS, you can use it.
@sahason Update: Thank you ! Your solution approach worked : Adoption of gsc.py and config templates helped to overcome those subscription related issues. But now we are blocked by another issue (which is most prob our internal problem ... ) :
Downloading metadata... error: cannot update repo 'rhel-9-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried; Last error: Curl error (60): SSL peer certificate or SSH remote key was not OK for https://cdn.redhat.com/customer/Library/content/dist/rhel9/9/x86_64/baseos/os/repodata/repomd.xml [SSL certificate problem: self-signed certificate in certificate chain]
I checked https://access.redhat.com/solutions/7066557 and informed our cloud support (Nevertheless any ideas from your end are very much appreciated )
@uwe-albert-ibm Please let me know the exact modification that you made to gsc.py and config templates?
@uwe-albert-ibm Please let me know the exact modification that you made to gsc.py and config templates?
@sahason Hi
in both config files we added COPY rhsm.conf /etc/rhsm/ COPY redhat-entitlement-authority.pem /etc/rhsm/ca/ COPY cert.pem /etc/pki/consumer/ COPY key.pem /etc/pki/consumer/
in gsc.py we added in the red-hat config method 4 copy statements which copy those 4 files mentioned above to the build directory (I am currently in a different vpn and send you the source snippet later)
@sahason Hi sorry I missed a 5th file which we copy
COPY rhsm.conf /etc/rhsm/ COPY redhat-entitlement-authority.pem /etc/rhsm/ca/ COPY cert.pem /etc/pki/consumer/ COPY key.pem /etc/pki/consumer/ COPY katello-server-ca.pem /etc/rhsm/ca/
and in gsc.py :: def handle_redhat_repo_configs(distro, tmp_build_path) we added at the end
shutil.copyfile('/etc/rhsm/rhsm.conf', tmp_build_path / 'rhsm.conf')
shutil.copyfile('/etc/rhsm/ca/redhat-entitlement-authority.pem', tmp_build_path / 'redhat-entitlement-authority.pem')
shutil.copyfile('/etc/pki/consumer/cert.pem', tmp_build_path / 'cert.pem')
shutil.copyfile('/etc/pki/consumer/key.pem', tmp_build_path / 'key.pem')
shutil.copyfile('/etc/rhsm/ca/katello-server-ca.pem', tmp_build_path / 'katello-server-ca.pem')
@uwe-albert-ibm Do you see any file in this folder /etc/pki/ca-trust/source/anchors/ on your host?
As per this ink https://access.redhat.com/solutions/7066557 that you share earlier what do you see under this section from this file /etc/rhm/rhsm.conf? [rhsm]
Content base URL:
baseurl= https://cdn.redhat.com
Server CA certificate location:
ca_cert_dir = /etc/rhsm/ca/
Default CA cert to use when generating yum repo configs:
repo_ca_cert = %(ca_cert_dir)sredhat-uep.pem
@sahason Hi sorry was in a call I ll check mom pls yes I see this katello key in the directory /etc/pki/ca-trust/source/anchors/ and the section in /etc/rhm/rhsm.conf? points to different base_url (some IBM link) and different key namely repo_ca_cert = %(ca_cert_dir)skatello-server-ca.pem
As described by red-hat one needs to replace those values, but we are already blocked when trying out the 2nd curl ... which is kind of prereq for replacing those 2 values ...
@sahason Hi sorry was in a call I ll check mom pls yes I see this katello key in the directory /etc/pki/ca-trust/source/anchors/ and the section in /etc/rhm/rhsm.conf? points to different base_url (some IBM link) and different key namely repo_ca_cert = %(ca_cert_dir)skatello-server-ca.pem
As described by red-hat one needs to replace those values, but we are already blocked when trying out the 2nd curl ... which is kind of prereq for replacing those 2 values ...
Copy the /etc/rhm/rhsm.conf as it is in the docekr image. Could you please let me know are you able to run these two curl command on host? Please replace the urls and cert path as mentioned in the /etc/rhm/rhsm.conf curl -v https://subscription.rhsm.redhat.com/subscription --cacert /etc/rhsm/ca/redhat-uep.pem curl -v https://cdn.redhat.com --cacert /etc/rhsm/ca/redhat-uep.pem
Are you referring to this curl command? curl -v https://cdn.redhat.com --cacert /etc/rhsm/ca/redhat-uep.pem What happen when you run the above command with IBM url and /etc/rhsm/ca/skatello-server-ca.pem
@sahason calling the curl with katello key (curl -v https://cdn.redhat.com --cacert /etc/rhsm/ca/katello-server-ca.pem) fails with curl: (60) SSL certificate problem: self-signed certificate in certificate chain More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.
the curl -v https://subscription.rhsm.redhat.com/subscription --cacert /etc/rhsm/ca/redhat-uep.pem works fine. whereas curl -v https://cdn.redhat.com/ --cacert /etc/rhsm/ca/redhat-uep.pem failes with http 403 (access enied)