arcade icon indicating copy to clipboard operation
arcade copied to clipboard

Move Maestro promotion pipeline from audit mode into enforcement

Open mmitche opened this issue 4 months ago • 0 comments

  • [ ] This issue is blocking
  • [ ] This issue is causing unreasonable pain

The Maestro promotion pipeline got some upgrades which prevent promotion of non-production builds to 'production' channels. There are quite a few audit failures right now. Most of these appear to be un-attributed production branches, but some are also cases where dev builds get assigned to 'production' channels. Many of these issues will get resolved when PRC gets turned on. The following script will check for audit failures:

$token = az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 | ConvertFrom-Json
$accessToken = $token.accessToken
$headers = @{ Authorization = "Bearer $accessToken" }

# Define pipeline targets
$targets = @(
    @{ Org = "dnceng"; Project = "internal"; PipelineId = 750 },
    @{ Org = "devdiv"; Project = "devdiv"; PipelineId = 12603 }
)

$results = @()

foreach ($target in $targets) {
    $buildsUrl = "https://dev.azure.com/$($target.Org)/$($target.Project)/_apis/build/builds?definitions=$($target.PipelineId)&api-version=7.1-preview.7"
    $builds = Invoke-RestMethod -Uri $buildsUrl -Headers $headers

    foreach ($build in $builds.value) {
        $timelineUrl = "https://dev.azure.com/$($target.Org)/$($target.Project)/_apis/build/builds/$($build.id)/timeline"
        try {
            $timeline = Invoke-RestMethod -Uri $timelineUrl -Headers $headers
        } catch {
            Write-Warning "Failed to retrieve timeline for build ID $($build.id)"
            continue
        }

        $barId = $null
        foreach ($tag in $build.tags) {
            if ($tag -match "BAR ID - (\d+)") {
                $barId = $matches[1]
                break
            }
        }

        if (-not $barId) {
            Write-Warning "No BAR ID found in build ID $($build.id)"
            continue
        }

        $found=$false

        foreach ($record in $timeline.records) {
            if ($record.issues) {
                foreach ($issue in $record.issues) {
                    if ($issue.type -eq "warning" -and $issue.message -like "*Build validation audit failure for production channel*") {
                        # Get DARC build info
                        $darcOutput = darc get-build --id $barId --output-format json | ConvertFrom-Json
                        if ($darcOutput.Count -eq 0) {
                            Write-Warning "No DARC build info found for BAR ID $barId"
                            continue
                        }

                        $buildInfo = $darcOutput[0]
                        $results += [PSCustomObject]@{
                            PipelineId = $target.PipelineId
                            BuildLink  = $buildInfo.buildLink
                            Branch     = $buildInfo.branch
                            Channels   = $buildInfo.channels -join ", "
                        }
                    }
                }
            }
        }
    }
}

$results | Format-Table -AutoSize

mmitche avatar Sep 09 '25 14:09 mmitche