RealOrangeOne
Status for aborted tasks
When using the database backend and stopping the worker, the worker does a graceful shutdown, but keeps the current task in the status "Running". As we are already doing a graceful shutdown, setting the task result to a new status like "Aborted" would make it more clear that the corresponding task is indeed not running any more.
If the shutdown is graceful, the task will be finished before terminating. Are you talking about if you try and kill the process again in a less-than-graceful way?
Apparently I enforced an actual kill with a second Ctrl + C, although another testing showed that the state would go into "Failed". So I guess this is more of a feature request or optional behavior: Instead of waiting for the task to finish, allow it to be aborted and the status set to aborted.
As some background: We have some heavier tasks (like license scans or aggregating statistics) which might take 10 minutes or sometimes two hours. Being able to abort them and have a suitable state would make identification/monitoring easier, while allowing for timely worker reloads.
I think there are two issues here:
-
Being able to abort a running task. There is currently no way to do that other than stopping the worker which leads to 2.
-
When the worker is stopped non-gracefully the task it was executing is stuck in RUNNING. Or rather there is a DBTaskResult with state RUNNING that is not executed anymore.
How do you stop (and restart) the worker in production where a user can't press Ctr+C?
(I might need to do that, too.)
How do you stop (and restart) the worker in production where a user can't press
Ctr+C?
At the moment, we are using a custom task system which we want to replace with some more "standard one". It basically uses systemd services running the workers. In theory, services can have something like TimeoutStopSec configured. To save the correct state, the current worker is implemented to have a stoppable background thread which we can ask to stop and might interrupt forcefully after some declared timeout. The actual implementation is more complex and might only work on Linux.