Improve dev productivity when working with Source Build release pipeline

[premun]

Goal

Improve velocity and confidence when making changes to the release pipeline.

Proposal

Following specific implementation features are proposed:

• Release metadata file –All inputs – both pipeline parameters and gathered information from other builds – are put into a JSON file which is published as a pipeline artifact.
  • This file then serves as input of further actions in the release.
  • It is easy to mock during PRs/dry runs where we don't have all inputs at hand.
  • It is possible to validate the file contents – we can validate the info gather part but also humans can view the file during the release to verify content when something is off.
• Three sets of pipelines – the pipelines differ only in how they receive inputs and what environment they target. Additionally, non-production pipelines could also clean up after.
  • PR build to validate changes before merging – it should run over both a 6.0/7.0 and a 8.0 flow.
  • Test pipeline (currently called dry-run) to run pipeline against a test environment – ideally should differ from official pipeline in the target environment only.
  • Official release pipeline only run for actual releases.
• YAML should go logic-free
  • All logic should be runnable locally. YAML only serves as a vehicle to process and flow inputs/data.
  • We should consider the PR run to validate that the inputs are relayed well (e.g. we don't use wrong YAML variable).
• Credential verification
  • The dry-run pipeline should have a stage for verifying credentials / connections to all production environment so that we can validate the release will happen without issues.

Concrete implementation ideas

• We should consider having a testable C# app with the logic that would be built and re-used in the pipeline.
  • It would have commands such create-release-announcement.
  • It would accept the JSON with inputs (it would have a command for creating it too).
  • It would have a dry-run mode (possibly set through env var) which would validate the expected inputs instead of real-world effects.
  • It would be more testable which could cover some blind spots during PR builds.
  • We should consider the ROI as many things the pipeline does are very easy to do in bash (which is not testable much though).
• Each step validates its effect so that we know the right thing happened. It's possible to run validation only too.
• We should set clear rules for when pipeline creates warning / errors. E.g. is it understandable when a dry run has warnings? etc…
• dotnet/source-build is onboarded onto secret manager so that environment access is kept functional.
• Consider the merger of the main release pipeline when working on this so that the two solutions are compatible.
• Consider having the release infrastructure internally only to allow better PR build tests.

[oleksandr-didyk]

Partially completed as part of https://github.com/dotnet/arcade-services/issues/2668

Implementation of a C# logic app for the pipeline will be complete as part of the Post 8.0 / Pre 9.0 effort

[premun]

Duplicate of #2668