Step Templates with high usage counts (1000+) will timeout when using "Update All..."

[donnybell]

Team

• [X] I've assigned a team label to this issue

Severity

Blocks affected customers, workaround listed below

Version

Likely present all current 2021.3 versions, reported in 2021.3.12132

Latest Version

No response

What happened?

When attempting to "Update All..." via the usage tab for a Step Template, SQL will timeout if the usage is too high.

Reproduction

n/a

Error and Stacktrace

See customer ticket

More Information

Customer Ticket [Internal]

Workaround

Use the PowerShell script below to work around the SQL timeout issue. This script will update all affected Steps automatically, but iterate one at a time:

$octopusUrl = "https://octopusURL" 
$octopusAPIKey = "API-XXXXXXXXXXXXXXXXXXXXXX" 
$header = @{ "X-Octopus-ApiKey" = $octopusAPIKey } 
$spaceId = 'Spaces-XXX' # The Spaces-XXX ID of the space (this can be found in the URL for any page on the space) 
$templateId = "ActionTemplates-XXX" # The ActionTemplates-XXX ID of the step template (this can be found in the URL of the step template, within Library -> Step Templates) 
$update = "" 

$json = Invoke-RestMethod -Method Get -Uri "$($octopusUrl)/api/$($spaceId)/actiontemplates/$templateId" -headers $header 
$latestVersionNumber = $json.version 
write-host "Latest Version number is $($latestVersionNumber)" 


$items = Invoke-RestMethod -Method Get -Uri "$($octopusUrl)/api/$($spaceId)/actiontemplates/$templateId/usage" -headers $header 
Foreach ($item in $items) { 
    if ($item.Version -lt $latestVersionNumber) { 
        $payload = @" 
            {“ActionsToUpdate":[{"ProcessId":"$($item.processid)",”ProcessType":"$($item.processtype)","ActionIds”:["$($item.actionid)"],”GitRef":""}],"Overrides":{},"DefaultPropertyValues":{},"Version”:"$($latestVersionNumber)"} 
"@ 
        $update = Invoke-RestMethod -Method Post -Uri "$($octopusUrl)/api/$($spaceId)/actiontemplates/$($templateId)/actionsUpdate" -header $header -body $payload 
        write-host "Updating step template on project $($item.processid) outcome: $($update.Outcome)" 
    } 
}

[donnybell]

Another affected customer [Internal Link] Step Template usage = ~500 and took ~15 minutes to update using the workaround script

[N-lson]

Most of the time is spent in handling the modify deployment store commands. Maybe we could simplify this? Replacing the mediator publish with a straight db update seems to produce the same result in 150s -> 7s:

await processStore.Update(deploymentProcess, cancellationToken);

But I haven't looked into why we're using the handler instead of directly calling the document store.

[brentkinney]

Another report of this issue https://octopus.zendesk.com/agent/tickets/96808 (internal link) Timeout occurred updating step template that was used in 300 projects

[donnybell]

Another affected customer [Internal Link]

[droyad]

We've had a look at this and attempted a fix, but quickly found that it is a complex problem. We're currently taking a step back and looking at how to best fix this.

[veochen-octopus]

Accidentally created a duplicate issue for this https://github.com/OctopusDeploy/Issues/issues/8203. I'll leave the release note on the other one.

[Octobob]

:tada: The fix for this issue has been released in:

Release stream	Release
2023.3	2023.3.4068
2023.4+	all releases