seastar
seastar copied to clipboard
scylladb
smp::submit_to performance problems
As shown in 0c2228a4916dc2dc66e9d587343ce436b92b7bef, on a busy target shard the smp::submit_to can take ~50ms to complete. Partially this is because a task that arrives from a smp message queue is put at the tail of the scheduler queue which is full of busyloops. However, in the example from the mentioned commit message the maximum time to traverse the queue should be nr_tasks * task_latency = 100 * 50us = 5ms.
Can't it take an arbitrary amount of time? If the task group in question is busy, it will be delayed.
Well, the test in question uses the single task group -- the default one -- and it is constantly busy. But my point is that since the arrived smp job is queued at the tail of the task group's queue, reactor should get to it once it scans this list at least once, i.e. in 5ms.
That's true for non-submit_to tasks as well, no? We have FIFO order usually.
Yes, but 5ms to process the task list should result in 1s/5ms=200 messages/s, but the result is 20 messages/s
@avikivity one more consideration that I discussed with @denesb today is that we don't guarantee aligned shard-to-shard connections for messaging, meaning that even for queries where the coordinator shard is communicating with the same shard on all replicas, there could be submit_to involved (on the remote end if I'm not mistaken). @gleb-cloudius please confirm.
@avikivity one more consideration that I discussed with @denesb today is that we don't guarantee aligned shard-to-shard connections for messaging, meaning that even for queries where the coordinator shard is communicating with the same shard on all replicas, there could be submit_to involved (on the remote end if I'm not mistaken). @gleb-cloudius please confirm.
There could be, but if all nodes have the same number of shard it is unlikely.
