Consider moving publishing infrastructure to another repo

[mmitche]

Currently publishing resides mostly in arcade, and runs as a service in the promotion pipelines. All .NET versions use this same problem (not bifurcated by release). This is by design. In addition, the release repo uses .NET 8 arcade, and uses the publishing tasks directly. This causes some annoyance because certain changes to publishing need to be backported to .NET 8.

It may make sense to move publishing to another repository altogether, and have staging and regular builds depend on the same set of functionality. In addition, it may even be better altogether for staging to depend on publishing as a service and not on the publishing tasks. Essentially, expand the feature set of the promotion pipeline to support what dotnet-release needs. This would ensure that we would not need to do an arcade update on channel updates.

There are a bunch of tradeoffs here that need to be evaluated. Moving publishing will solve some issues at the expense of others. Key points and general considerations:

• Motivations based on improving staging's lifecycle should wait until extended .NET 6 support ends. Any move of staging to use a newer arcade needs to also evaluate the other changes in arcade that have happened in the meantime. The key bit here is that the post build signing logic has languished in main. It is ONLY used in 6 and we are trying to rip it out altogether.
• Publishing is not separate from repo builds. There is a 'contract', but simplifying publishing will require changes to that contract. A repo build's manifest generation, artifact metadata etc. is heavily integrated with the actual publishing logic. Splitting into another repo will make the dev cycle for those changes significantly longer and more difficult.
• Publishing logic is also integrated into the YAML - Some publishing logic must live in the YAML files. Specifically calls to Maestro, artifact handling, etc. This might be moveable into another repo and used via templates, but be wary of designing an overly complex system.
• Some kinds of publishing are integrated into the repo build. Legacy symbol publishing functionality is still active in some repos.
• Publishing is heavy on the SFI. Maintaining multiple copies is expensive. Avoid branching publishing for servicing.

I favor the following approach, if we are to move publishing out of arcade:

• Clean up old contracts, variables, etc. to ensure we know what the active code paths are.
• Iterate on an updated contract + publishing functionality (probably v4?) in main arcade.
• When complete, backport the repo build side of that contract to older versions as necessary.
• Rip out v3 publishing and vestiges of old contracts in arcade, obsolete repo YAML, etc.
• Move publishing out of arcade and arcade-services into another repo.

One of our biggest sins was that this step did not get done last time.