option to turn off publishing assets in parallel

[moltar]

It is a bit annoying and error-prone to re-generate the GitHub Actions file every time there is a new asset is added. Which can be ambiguous as many constructs add assets under the hood.

Is there a way to combine the assets into a single job?

CodePipeline has that option.

[kaizencc]

Hi @moltar, thanks for opening the issue. I'm assuming by GitHub Actions file you are referring to the deploy.yml workflow file. It's definitely an interesting concept but I'm not convinced that this is the behavior we should go down.

Is there a way to combine the assets into a single job?

I don't think there's anything blocking this behavior, though it would be a large refactor. Basically, instead of writing a file per asset, we would write one well-known file containing instructions for deploying every single asset and link that to deploy.yml. But then suddenly we'd have to keep track of this well-known file along with deploy.yml. There's also all sorts of weird issues that arise when you're expecting deploy.yml not to change but the pipeline behavior to change.

CodePipeline has that option.

The feature parity we strive for is for cdk-pipelines-github to be functionally equivalent to @aws-cdk/pipelines whenever possible. AFAIK, @aws-cdk/pipelines publishes each asset individually, but I could be wrong. Can you link me to the behavior you see in CodePipeline, or better yet the equivalent behavior in @aws-cdk/pipelines?

[moltar]

Hi @kaizencc, appreciate your detailed response!

The prop I am talking about is publishAssetsInParallel.

Docs: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.pipelines.CodePipeline.html#publishassetsinparallel

asset and link that to deploy.yml. But then suddenly we'd have to keep track of this well-known file along with deploy.yml

Why can't we use cdk.out manifests for tracking the assets? If this goes into the artifact, and then come out of artifact all intact this should work. I mean it works in the CodeBuild setup, and it doesn't have any capacity to save state between executions, besides artifacts as well. So if it works in CB, should work with GHA too. No?

[kaizencc]

Why can't we use cdk.out manifests for tracking the assets?

We don't actually track the contents of cdk.out and recommend you not commit this folder to your repository. What should be happening is that the synth step of your pipeline will create a cdk.out and that is the assembly that gets deployed. It becomes problematic if the same deploy.yml file can produce different cdk.out folders imo.

Regarding publishAssetsInParallel, I see your point. The default behavior of true is the only behavior that is allowed in cdk-pipelines-github. I suppose it is reasonable to expect a way to not publish assets in parallel via github actions. However, I don't think introducing something similar in this module would mean that deploy.yml doesn't change when you introduce a new asset to the pipeline. In this scenario, sure there's only one "publish asset" step but deploy.yml still updates to a new publish-assets-<hash>.sh file. I think you want deploy.yml to change every time your pipeline changes.

Updating this issue to reflect the feature request of a publishAssetsInParallel option, though I don't think it's on my immediate roadmap without a few additional +1s.

[moltar]

and recommend you not commit this folder to your repository.

Of course, but that's not what I meant.

I meant tracking the assets via artifacts in GH.

1. Build generates cdk.out
2. Adds it to artifact
3. Next step downloads the cdk.out artifact
4. Read manifest file for available assets
5. Upload all assets in one step

[kaizencc]

Yep. I agree with all that. What I disagree with is keeping the manifest file "well-known", so that deploy.yml does not change. That will introduce unintended consequences imo. I think this works if the manifest file name is hashed, so deploy.yml will target a different file name whenever the contents change. You'll get your assets in one step, but deploy.yml will still change every time your assets change.

I'll think this through a bit more, but those are my thoughts right now.

[moltar]

I don't understand tho why manifest needs to be well known?

To my understanding, cdk can work with cdk.out alone. That's how CodePipeline via AWS works now, I think.

Once you do cdk synth and package up the cdk.out, that is all that is needed to have as the final artifact to be able to do cdk deploy.

npx cdk deploy --app "cdk.out/"

[maryal-iag]

Did this go anywhere? It would great if all the assets could be uploaded in a single job rather than multiple in parallel. We are getting pipeline which look like the following and it is spawning a separate runner for each publish...