yii2-queue
yii2-queue copied to clipboard
yiisoft
TTR Does not work as advertised.
What steps will reproduce the problem?
Set up any queue with TTR 30. Push a job that does this:
for ($i = 0; $i < 1000; $i++) {
echo $i . "\n";
sleep 1;
}
Run 2 queue runners.
What's expected?
Based on the docs:
The TTR option sets the time to reserve for job execution. If the execution of a job takes longer than this
time, execution will be stopped and it will be returned to the queue for later retry.
What do you get instead?
After the 30 seconds have passed the job is released by the queue (beanstalkd). The second worker picks it up and starts executing it. The first worker continues to execute it.
There are 2 solutions:
Proposal 1: Fix docs
Fixing the documentation will make it work as expected.
Proposal 2: Fix functionality
There are 2 methods of executing jobs, in-process and isolated process. In the case of using an isolated process the solution is simple: kill it just before the TTR expires. In case of in-process execution I have created a proof of concept to handle the TTR:
<?php
declare(ticks=1);
$end = microtime(true) + 5;
$handler = function() use ($end) {
$remaining = $end - microtime(true);
echo ".";
if ($remaining < 0) {
throw new \Exception('time limit exceeded');
}
};
register_tick_function($handler);
try {
$i = 0;
while ($i < 1000) {
// This handler suppresses errors.
try {
echo "\$i is $i\n";
sleep(1);
$i++;
} catch (\Throwable $t) {
// echo "E: {$t->getMessage()}\n";
}
}
} catch(\Throwable $t) {
unregister_tick_function($handler);
echo "Caught: {$t->getMessage()}\n";
} finally {
unregister_tick_function($handler);
}
..$i is 0
....$i is 1
....$i is 2
....$i is 3
....$i is 4
...Caught: time limit exceeded
The idea is simple:
- Register a tick function that keeps track of time.
- As soon as time is over we start throwing exceptions.
- Job code might be catching exceptions, but as soon as their try clause is called we throw another one (this is repeated for nested
try..catch). - Finally the exception reaches our own
catchclause where immediately unregister the tick function.
This way we can guarantee that no 2 workers are executing the same task at any point in time.
@samdark This is not driver specific. There is no driver that limits the execution time of a job. In some drivers (at least beanstalk and db) it results in the job being released to multiple workers.
@SamMousa there Symfony/Process is used and interrupts after timeout (TTR). Just tested it - works as expected.
https://github.com/yiisoft/yii2-queue/blob/master/src/cli/Command.php#L183 https://github.com/symfony/process/blob/master/Process.php#L140
What OS you use ?
@acidka I'm on linux, but I don't use the out of process execution so If you say it works there then I believe you, I didn't check.
I just noted that if it isn't supported there, that case is easy to implement. For the in-process execution it never works.
@SamMousa it is because in-process execution runs directly without any control just in try-catch https://github.com/yiisoft/yii2-queue/blob/master/src/Queue.php#L214
Yes, I know why it is, I'm just proposing to fix that either by fixing the functionality or by fixing documentation
The "touch" command allows a worker to request more time to work on a job.
This is useful for jobs that potentially take a long time, but you still want
the benefits of a TTR pulling a job away from an unresponsive worker. A worker
may periodically tell the server that it's still alive and processing a job
(e.g. it may do this on DEADLINE_SOON). The command postpones the auto
release of a reserved job until TTR seconds from when the command is issued.
The touch command looks like this:
touch <id>\r\n
- <id> is the ID of a job reserved by the current connection.
There are two possible responses:
- "TOUCHED\r\n" to indicate success.
- "NOT_FOUND\r\n" if the job does not exist or is not reserved by the client.
https://github.com/beanstalkd/beanstalkd/blob/master/doc/protocol.txt#L328-L345
