Add some safe guards for the Matrix of Truth

[dkurepa]

Currently we have no real safeguards around the Matrix of Truth. If someone makes some breaking changes, they could also break the power BIs. We need either a staging environment, or a process to handle these

[dkurepa]

I've talked about this with Ricardo and we think we have a good ideas:

• First would be to run the real Matrix of Truth pipeline off of the production branch (currently it is run off of artifacts from the latest successful main build). We would also have a separate, staging, pipeline that would run every time there's a change to the MoT code. There would be a staging Power BI report that would consume the outputs produced by the staging pipeline. Ideally, we would download the staging Power BI reports in some machine consumable format and able to somehow, automatically check them. If this is not possible, we could make checking of the staging Power BI reports a part of the helix-machines rollout. The person responsible for the rollout would have a look at the report produced from the latest changes and see if everything is OK. This does have a downside, because a person would have to know how the report should look, but I don't think that will be an issue.
• We've also talked about having a pipeline run off of a PR branch, generating the outputs in a staging storage account, which would be consumed by a staging power BI report and then looking then and there, but this has a problem with race conditions. If more than one person is working on the MoT and they have PR's in the same time, the first persons Pr's would be overwritten

[dkurepa]

pr: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines/pullrequest/24714 This pr adds staging outputs that are uploaded to a staging storage account, which are then consumed by the staging Power BI (not created yet). These outputs are created on every main build.

[dkurepa]

Updating the file one last time: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines/pullrequest/25920

[ilyas1974]

A report has been created that targets the data within the staging storage account. Whenever a change is made to the Matrix of Truth staging environment, we simply need to access this report, refresh the data and make sure the pages render properly with whatever changes have been made before rolling out to production.