Optionally record additional information about `cosa buildupload s3` location + access to that location in meta.json

[miabbott]

Feature Request

We wish to be able to upload the results of a successful RHCOS build into Brew. However, to avoid the huge penalty of uploading every (uncompressed!) artifact into Brew, we've reached an agreement with the Software Production folks that providing metadata about the build should suffice.

Since we treat S3 as our canonical source of truth (the bucket we use contains every production RHCOS build ever made), we want to be able to record this bucket in meta.json as a way for other processes that operate on Brew builds to know where the uploaded artifacts are located. Additionally, since the bucket is private and not accessible publicly, we want to provide information about the redirector service (https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/) so that it is possible for users to determine how to access the artifacts in the S3 bucket, by way of the redirector.

Desired Feature

The original design that I'm proposing here consists of two parts: an update to the meta.json schema and an enhancement to the cosa buildupload command.

The update to the meta.json schema would include a new optional s3 dict that would allow us to record the bucket name, subdirectory/path, and URL for the redirector service.

The enhancement to the cosa buildupload command would be some new option(s) that would allow the user to specify that information about the S3 bucket, path, and URL should be recorded in meta.json. Perhaps these options are only applicable to cosa buildupload s3.

Example Usage

Using the existing mechanism, this would instruct the command to split the bucket/path from the final argument and then record all the info to meta.json

$ cosa buildupload s3 --acl=private --record-s3-metadata --s3-url https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/art/storage/releases/rhcos-4.7-ppc64le art-rhcos-ci/releases/rhcos-4.7-ppc64le

Alternatively, change the invocation to support explicit options to specify bucket, path, url:

$ cosa buildupload s3 --acl=private --s3-url https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/art/storage/releases/rhcos-4.7-ppc64le --s3-bucket art-rhcos-ci --s3-path releases/rhcos-4.7-ppc64le

Other Information

This is in support of making RHCOS artifacts available to the various cloud marketplaces. See the AWS Marketplace Epic and the associated work stories attached to it https://issues.redhat.com/browse/COS-6

There is an outstanding PR that started on some of this request - https://github.com/coreos/coreos-assembler/pull/2731

[jlebon]

Nice write-up!

The request makes sense to me. My only minor hesitation with the suggested approach is having the canonical meta.json encode a location to itself. It would become stale if we moved buckets or (more likely) if the build dir was copied to another bucket/location.

Hmm, would it be sufficient to just put that information in a separate file when uploading to Brew? It doesn't solve the stale issue, but at least it makes it less likely that something else would use that information.

But not strongly against shoving it in meta.json either.

[cgwalters]

Worth keeping in mind that https://github.com/coreos/coreos-assembler/issues/2685 is pushing to change our "center of gravity" to be more container focused and less S3.

[miabbott]

The request makes sense to me. My only minor hesitation with the suggested approach is having the canonical meta.json encode a location to itself. It would become stale if we moved buckets or (more likely) if the build dir was copied to another bucket/location.

Hmm, would it be sufficient to just put that information in a separate file when uploading to Brew? It doesn't solve the stale issue, but at least it makes it less likely that something else would use that information.

I'm less concerned with the problem of stale information in this situation. Broadly speaking, if the build dir is being copied to a new location, the person/robot doing so should be aware of the implications and not be relying on the information about the S3 bucket after the copy operation.

That being said, I wasn't terribly comfortable with encoding this data in meta.json as it doesn't seem to be an exact fit, so recording the info in a separate file may be fine. I don't think the folks operating on the Brew builds will mind, as long as we have the new file and new metadata well defined/described.

Since @ravanelli is already going forward with the meta.json implementation in #2731, I'm not going to argue hard for a separate file.