Add `pipeline.yaml`

[cgwalters]

This is motivated by https://github.com/coreos/fedora-coreos-pipeline/issues/618

Today the coreos-assembler default for cosa compress is gzip, and the RHCOS pipeline relies on that. The FCOS pipeline overrides it.

My opinion is that this is exactly the kind of thing that should be configured in the source git - it should not live in "the pipeline".

By having a new pipeline.yaml in this repository, we get closer to the goal of having the default invocations of cosa $x do the same thing as production pipelines.

If accepted I plan to extend this with numerous other things; most notably for example the set of desired target artifacts i.e.

artifacts:
  - aws
  - metal
  - digitalocean
  - ...

etc. Then we can have cosa buildextend-all, etc.

This PR includes some ugly bridging of Go logic to Python.

[cgwalters]

(Potentially we should have exclude-artifacts instead; that could work well)

[dustymabe]

I'm not opposed to storing some more information in the src/config. Obviously @jlebon knows this space very well and I'd be interested in his thoughts here.

[cgwalters]

I'm not opposed to storing some more information in the src/config. Obviously @jlebon knows this space very well and I'd be interested in his thoughts here.

OK, yes we can debate next week. But here's another way to look at this:

The pipeline unification and things like https://github.com/jlebon/fedora-coreos-pipecfg/blob/main/config.yaml are good.

But again - even after that lands and is working 100% perfectly, we will still have other "pipelines" - things like https://github.com/openshift/os/blob/master/ci/prow-entrypoint.sh and the CI flows across repositories (some of which use Jenkins, some don't). And even beyond that the SCOS folks are now orchestrating cosa with Tekton: https://github.com/okd-project/okd-coreos-pipeline

I think this type of thing along with the desire to have typing cosa $x Do The Right Thing (as argued above) combines to put a strong pressure to have something like this PR where we tie as much as we can to the config git in a format that is not in any way tied or related to Jenkins.

[jlebon]

While working on https://github.com/coreos/fedora-coreos-pipeline/pull/619, I did briefly think about something like this too. More specifically, I was thinking about having the config repo declare which features it supports, e.g. something like:

# features.yaml
platforms:
- nutanix

Imagining when Nutanix support was just added, this file would correctly get promoted/branched during the FCOS/RHCOS promotion process with the supporting code so that we don't actually build Nutanix images for streams which don't support it. And once it becomes supported everywhere, we could remove it from the file.

This is similar to this PR but subtly different: the features.yaml is conceptually more like metadata for what the distro supports as a whole and isn't trying to describe a pipeline. But it could of course be used by the pipeline. E.g. we could have a cosa buildextend-all use this and knob defaults for cosa in it (though specifically for artifact compression, it seems like that could just live in image.yaml, no?).

I'm not convinced about having a more pipeline-focused file in the config repo. Conceptually, I consider pipelines a layer of abstraction above the config repo. You could have multiple pipelines using the config repo for different purposes, and they might need to build it differently (signing vs no signing, with some overrides, minimal vs all artifacts, uploading vs no uploading, etc...) or in different stages. I think this is already true today.

[jlebon]

That said, having support for a tool which takes a pipeline.yaml and runs the appropriate cosa subcommands is definitely an interesting avenue (i.e. as discussed in https://github.com/coreos/coreos-assembler/issues/3008).

[cgwalters]

Rebased :surfer: and fixed the f-string lint thing.

[cgwalters]

I know there's a lot going on, but conceptually this doesn't conflict with anything today and I think will be useful.

[cgwalters]

Rebased :surfer:

[cgwalters]

OK, I still think this makes sense to land. I need to make things such that we inject metadata like this:

             io.openshift.build.version-display-names=machine-os=Red Hat Enterprise Linux CoreOS
             io.openshift.build.versions=machine-os=412.86.202210211031-0

that's on the previous/legacy oscontainer on our new one. I think it'd be done most elegantly via this config file. If we configure it code linked to the Jenkins pipeline, then cosa build will do the wrong thing locally.

I guess I can stick it in image.yaml for now...

[jlebon]

Just opened https://github.com/coreos/coreos-assembler/pull/3215 for the compression bit specifically since I think that's a good fit for image.yaml.

DO you have any thoughts about https://github.com/coreos/coreos-assembler/pull/3094#issuecomment-1260082345?

[openshift-merge-robot]

@cgwalters: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jlebon]

The concept of a pipeline.yaml could still make sense, even as we move to a Tekton-based world. But (1) I think it probably shouldn't live in the src config repo, and (2) it's possible that even in Tekton/other orchestrators, it's not as straightforward to have a single pipeline invocation that cosa can consume. E.g. you might have intermediate steps (signing), or want better visualization in the orchestrator's view, or have different resources requirements for different stages (e.g. testing and compression have wildly different resource requirements from uploading). Closing this for now, but we can rediscuss the idea in the future as we do e.g. the Tekton move.