Issues icon indicating copy to clipboard operation
Issues copied to clipboard

Unresolved manual intervention holding transaction open forever

Open veochen-octopus opened this issue 2 years ago • 2 comments

Team

  • [X] I've assigned a team label to this issue

Severity

kinda non-blocking

Version

Reported on 2021.2.7650

Latest Version

No response

What happened?

Unattended manual interventions after some time cause db (replicated) to time out. The tempdb version stores grow too large. We believe there's a transaction somewhere being kept open while waiting for the interruption to resolve.

Reproduction

Below is a best guess

  • Create a project with a simple deployment process that would error
  • Use guided failure
  • Deploy, which should result in a manual intervention
  • Wait for a long time and watch the tempdb version stores grow in size

Error and Stacktrace

No response

More Information

Internal slack discussion - https://octopusdeploy.slack.com/archives/CNHBHV2BX/p1656639072790969 More discussion - https://octopusdeploy.slack.com/archives/CNHBHV2BX/p1660868210247089 Customer report - https://octopus.zendesk.com/agent/tickets/92111

Workaround

Customer suggests restarting the mssql services resolves the issue at the cost of losing the intervention task, so is not a full workaround.

veochen-octopus avatar Jul 04 '22 04:07 veochen-octopus

Another affected customer: https://octopus.zendesk.com/agent/tickets/92598

donnybell avatar Jul 08 '22 18:07 donnybell

Another customer is facing this error: SQL Error 1222 - Lock request time out period exceeded Their issue could be related to this manual intervention problem: [internal] ticket.

akirayamamoto avatar Aug 19 '22 00:08 akirayamamoto

Fixed in: 2023.1.9794 2022.4.8583 2022.3.11320

evolutionise avatar Apr 11 '23 22:04 evolutionise

Release Note: Fixed an issue where a long-running database transaction would grow the transaction log unreasonably large during a long deployment

octoreleasebot avatar Apr 11 '23 23:04 octoreleasebot

:tada: The fix for this issue has been released in:

Release stream Release
2022.3 2022.3.11320
2022.4 2022.4.8583
2023.1 2023.1.9794
2023.2 2023.2.5159
2023.3+ all releases

Octobob avatar May 19 '23 07:05 Octobob