coreos-assembler
coreos-assembler copied to clipboard
coreos
src/cmd-init: add a way to import protected .repo files and content sets
Add a new option --extrarepos to allow import of .repo files and content sets
located outside of GITCONFIG.
Motivation: RHCOS needs yumrepo definitions hidden behind a firewall. Let's add a clean way to import them.
Hmm. I'm OK with this, but IMO it also cuts against the "elegance" of the current architecture. For example, suddenly pull requests don't quite work because there are two git repositories involved. We're also today injecting metadata about the config git repository into the base container image. But that loses some meaning with two git repositories.
Related to that topic, this PR isn't injecting any metadata from (e.g. exact git rev) the "extconfig" git repo into the cosa metadata.
Hmm. I'm OK with this, but IMO it also cuts against the "elegance" of the current architecture. For example, suddenly pull requests don't quite work because there are two git repositories involved. We're also today injecting metadata about the config git repository into the base container image. But that loses some meaning with two git repositories.
The use case for this is RHCOS, which sadly already suffers from this issue. The idea with this PR is essentially upstreaming and formalizing the cp hack that currently exists in the RHCOS pipeline into cosa.
Related to that topic, this PR isn't injecting any metadata from (e.g. exact git rev) the "extconfig" git repo into the cosa metadata.
:+1: Yeah we should add that (and is actually another benefit of formalizing it in cosa directly).
Note: if we have lockfiles, then this extconfig repo degrades basically to a mirrorlist.
I think we should coordinate the changes in this one with those from https://github.com/coreos/coreos-assembler/pull/2934
The only concern I could find was about the additional repo becoming stale on developer systems. This will not be an issue in CI as we clone from scratch every time.
One option to work around that is to include the minor RHEL version in the repo names thus the build would fail when the repos are missing.
Thanks. This looks good to me. I would prefer another name for the option (
repo,add(itional)-repo,private-repo,extra-repo?) but this is close to a bikeshed discussion so not blocking on that.
Yeah, it's tricky. The main issue I was working against was that repo is a very overloaded term. I called it --yumrepos to clarify that's the type of files we get from it, but the metavar GITREPO to clarify that we expect a git URL.
I can imagine renaming it to something more generic if we start copying more than yum repos and content sets. But hopefully we never do that. As mentioned above, ideally we'd be able to keep everything in openshift/os (I found https://github.com/AGWA/git-crypt which looks interesting though storing encrypted files in a public repo doesn't have great optics).
The only concern I could find was about the additional repo becoming stale on developer systems. This will not be an issue in CI as we clone from scratch every time.
Good point. The same concern applies to the primary config repo. In a follow-up, I'll add support for symlinking. This will allow devs to point to their existing checkouts just like for the config repo, which should help counter that a bit.
One option to work around that is to include the minor RHEL version in the repo names thus the build would fail when the repos are missing.
We can't change the repo names unfortunately because they're used in the content set mapping.
In a follow-up, I'll add support for symlinking. This will allow devs to point to their existing checkouts just like for the config repo, which should help counter that a bit.
Done in https://github.com/coreos/coreos-assembler/pull/3045.
This likely broke RHCOS CI: https://github.com/openshift/os/pull/959. Investigating:
+ cp src/config/repos/c9s.repo src/config/c9s.repo
+ curl -L http://base-4-12-rhel90.ocp.svc.cluster.local -o src/config/tmp.repo
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 804 100 804 0 0 100k 0 --:--:-- --:--:-- --:--:-- 112k
+ awk '/rhel-9-server-ose/,/^$/' src/config/tmp.repo
+ echo includepkgs=cri-o,cri-tools,openshift-clients,openshift-hyperkube
+ rm src/config/tmp.repo
+ cosa fetch
Config commit: v0.0.2-785-gf2b445d4302a85fc157526e766fe9c209158dbba
Using manifest: /tmp/cosa/src/config/manifest.yaml
ls: cannot access '/tmp/cosa/tmp/override/*.repo': No such file or directory
ERROR: no yum repo files were found
error: failed to execute cmd-fetch: exit status 1
And I've realized late that this will also not work when we have two minor RHEL minor versions at the same time in the repo (9.0 & 9.2) as the repo names will be the same but not the URLs.
We thus have to make repo names minor version specific (this should also work for content sets) or rework this one.
We also need to be able to specify a branch for the yumrepo git repo, unless we make all repo names explicitly include their version.
We thus have to make repo names minor version specific (this should also work for content sets)
I'm in favor of this.
And I've realized late that this will also not work when we have two minor RHEL minor versions at the same time in the repo (9.0 & 9.2) as the repo names will be the same but not the URLs.
We thus have to make repo names minor version specific (this should also work for content sets) or rework this one.
Yes, if the repo contents are different, we shouldn't be using the same repo ID. Otherwise we'll be claiming we're using the wrong content set for one of them. This is what this comment is about.
And this jogs my memory now. I said higher up:
We can't change the repo names unfortunately because they're used in the content set mapping.
This is not true. I was thinking instead I think of the fact we have to match the repo names of the mirrored versions in OpenShift CI, which I think we have some control over.
Yes, we have control over the names in Prow CI. I'll make the PRs to change everything.