Non-determinism in publishing in presence of errors

[mmitche]

• [ ] This issue is blocking
• [ ] This issue is causing unreasonable pain

There is possible non-determinism in publishing. We saw some cases in staging where packages that were not found during publishing caused blob publishing to not happen at all. While there are still errors, they should not cause non-deterministic behavior if possible. Here is what I think is going on:

So I think basically what is happening is this:

• All the publishing is happening in parallel
• Very early on, the tasks that are handling the package publishing notice that fsharp files are missing and they file a complaint with MSBuild arcade/PublishArtifactsInManifestBase.cs at main · dotnet/arcade (github.com)
• The LogError affects the overall task state (HasLoggedErrors)
• The blob publishing hits the "HasLoggedErrors" that is set by the package publishing before it gets a chance to start actually pushing things: arcade/PublishArtifactsInManifestBase.cs at main · dotnet/arcade (github.com)

Errors should still be reported, but we should ideally avoid non-deterministic behavior. All of the publishing lanes (blobs, packages, and symbols) go in parallel, and there is parallelism within those tasks themselves (publishing multiple at once). Publishing is highly optimized.

The right thing to do to avoid use of the mutable global state (Log.HasLoggedErrors) until all parallel tasks have been resolved. Instead, introduce appropriate non-global state and check that.