queue
queue copied to clipboard
tube: fix slow take on busy utubes
If some of the utubes for tasks at the start of the queue were busy most of the time, take
would slow down for every other task. This problem is fixed by creating a new space space_ready
. It contains first task with READY
status from each utube.
This solution shows great results for the stated problem, but with the cost of slowing the put
method (it is ~3 times slower). Thus, this workaround is disabled by default. To enable it, user should set the v2 = true
as an option while creating the tube. As example:
local test_queue = queue.create_tube('test_queue', 'utube',
{temporary = true, v2 = true})
Also added two benchmarks to compare v2 = false
and v2 = true
(v2 = false
is default and disables every change from this PR).
- Benchmark for simple
put
andtake
methods. 30k utubes are created with single task each. Task creation time is calculated. After that 30k consumers are callingtake
+ack
, each in the separate fiber. Time to ack all tasks is calculated. The results are as follows (for comparison I also usedfifo
tube):As we can see, new utube implementation has ~3 times slower
put
method. - Benchmark for the stated problem. 10 tubes are created. Each contains 1000 task. After that 10 consumers are created (each works on his tube only, one tube -- one consumer). 5 of them will take a single task and end their execution without acking the task. This way we will have 5
TAKEN
utubes. Other 5 consumers will justtake
+ack
every task from their utube (1000 tasks each). After that we can also run this benchmark with 10k tasks on each utube, 100k tasks and 150k tasks. But all that with 10 utubes and 10 consumers. The results are as follows:We can see great time performance improvement.
Same change was also made for utubettl
. Here is benchmarks results for utubettl
(on the same benchmarks):
(note that for
busy tasks
were used 140k and not 150k tasks).
Closes #228
Right now only utube
is fixed. utubettl
will follow soon.
Added fix for utubettl
in a different commit.
@DifferentialOrange @better0fdead this is a non-trivial fix, we need additional reviews with a clear head here.
Updated the code according to comments.
Added links to the new issue https://github.com/tarantool/queue/issues/230 about implementing new storage modes for vinyl
engine.
Updated the code according to the comments.
Updated the code according to comments.