Scheduled Triggers Not Firing - ServerTask Stuck in Queued State - SQL Timeout

[FinnianDempsey]

Severity

workaround exists but no alerting on issue to know when to action

Version

2024.1.12722

Latest Version

I could reproduce the problem in the latest build

What happened?

The Process recurring scheduled triggers ServerTask is able to get stuck in the Queued state causing triggers to no longer work.

This queued task will remain until the Octopus Server node is restarted.

Reproduction

Tricky to reproduce, possibly due to SQL timeout

Error and Stacktrace

N/A - Only symptom is the ServerTask stuck in queued state indefinitely

More Information

Internal Link - Slack

Workaround

Caution!

When implementing one of the workarounds below, all scheduled triggers will "catch up" from when the task became stuck. This can cause deployments, tasks, and runbooks to run unexpectedly.

--->Please use caution and monitor the task queue once the block is removed! <---

Restart the Octopus node or use the workaround from https://github.com/OctopusDeploy/Issues/issues/8485, found below:

This issue is caused by a task stuck in the queue, which prevents new tasks being added. The issue can be resolved by cancelling the stuck task. This must be done via the API.

Step 1: Find the stuck task

Use the following command to find the stuck task.

curl "https://<SERVER_URI>/api/tasks?spaces=all&states=New&name=RecurringScheduledDeployments" -H "X-Octopus-ApiKey: <API_KEY>"

This should return a response like this:

{
  "ItemType": "Task",
  "TotalResults": 1,
  "ItemsPerPage": 30,
  "NumberOfPages": 1,
  "LastPageNumber": 0,
  "TotalCounts": {
     ...
  },
  "TotalCountsInOtherSpaces": {
     ...
  },
  "Items": [
    {
      "Id": "ServerTasks-432",
      "SpaceId": "Spaces-1",
      "Name": "RecurringScheduledDeployments",
      "Description": "Process recurring scheduled triggers",
      "Arguments": {
        "ProcessingDate": "2023-11-22T05:31:29.734"
      },
      "State": "Queued",
      "Completed": "Queued...",
      "QueueTime": "2023-11-22T05:31:29.769+00:00",
      "QueueTimeExpiry": null,
      "StartTime": null,
      "LastUpdatedTime": "2023-11-22T05:31:29.770+00:00",
      "CompletedTime": null,
      "ServerNode": null,
      "Duration": "1 minute",
      "ErrorMessage": "",
      "HasBeenPickedUpByProcessor": false,
      "IsCompleted": false,
      "FinishedSuccessfully": false,
      "HasPendingInterruptions": false,
      "CanRerun": true,
      "HasWarningsOrErrors": false,
      "UnmetPreconditions": null,
      "Links": {
         ...
      }
    }
  ],
  "Links": {
     ...
  }
}

In this case, the stuck task is ServerTasks-432 in Spaces-1.

Step 2: Cancel the stuck task

Use the below command to cancel the task. This example uses the IDs from the previous step.

curl -X POST "https://<SERVER_URI>/api/Spaces-1/tasks/ServerTasks-432/cancel" -H "X-Octopus-ApiKey: <API_KEY>" -d ""

With the stuck task cancelled the triggers should start running again after a minute or so.