docker-tools icon indicating copy to clipboard operation
docker-tools copied to clipboard

Split build/test and publish stages into separate templates

Open lbussell opened this issue 9 months ago • 3 comments

This is needed for https://github.com/dotnet/dotnet-docker-internal/issues/7413, because we need to call the publish stage separately from the build/test stages. I took the simplest approach, which is just making the build and publish stages their own templates, with their own inputs.

~~build-test-publish-repo.yml still contains some important default values, so the next step would probably be to push the default values down into the sub-templates. We can also pull the giant nested conditional out of publish.yml back up into build-test-publish-repo.yml, because that logic assumes that you aren't running the publish stage by itself.~~

Update after https://github.com/dotnet/docker-tools/pull/1652/commits/0a151ba6a395e2552c59c12841a82e8985e7e84f - Doing fancy stuff with pipelines was taking too long, so this PR just keeps it simple. The templates could be cleaned up a little more but everything is functional for now.

lbussell avatar Mar 19 '25 17:03 lbussell

I marked this as draft for now because I need a little more time to settle on a better pattern.

lbussell avatar Mar 24 '25 15:03 lbussell

I tested this by queueing a build of dotnet-docker's nightly-playground pipeline with these changes (as well as additional changes needed for that repo). I compared the generated yaml pipeline to what was generated for the current nightly branch. They are line-for-line identical (all 15k lines).

lbussell avatar Mar 24 '25 22:03 lbussell

FYI @dagood, minor breaking change here.

lbussell avatar Mar 24 '25 23:03 lbussell

It's not really clear to me why this is necessary (vs. queueing the pipeline without build in stages (or something similar with a parameter?)) but regardless, it seems easy enough for us to migrate.

Well... because I didn't think of that 😄

Separately, I don't really like controlling the stages using those variables/conditions because it seems fragile to me - esp. because variables are mutable and can be set by any step's output. Since we're dealing with pre-release images now I would rather the publish stage be excluded completely at queue time. So that's probably why that approach didn't cross my mind to begin with.

We could add another wrapper similar to the dotnet/build-test-publish-repo.yml one to make this a completely non-breaking change.

lbussell avatar Mar 25 '25 18:03 lbussell

We could add another wrapper

But we can do that in a separate PR if necessary.

lbussell avatar Mar 25 '25 18:03 lbussell

Separately, I don't really like controlling the stages using those variables/conditions because it seems fragile to me - esp. because variables are mutable and can be set by any step's output. Since we're dealing with pre-release images now I would rather the publish stage be excluded completely at queue time. So that's probably why that approach didn't cross my mind to begin with.

Ah, reasonable. In Go pipelines I avoid using variables and conditions entirely. 😄 (I specifically don't like that time after queueing where you don't know if certain stages/jobs/steps will run or not.) Splitting the yml conceptually seems like the same direction--more at template eval time rather than runtime--and I support that.

dagood avatar Mar 25 '25 19:03 dagood