restate icon indicating copy to clipboard operation
restate copied to clipboard

Update shuffle to use tailing iterator for performance improvements

Open tillrohrmann opened this issue 1 year ago • 1 comments

With the fix for #1639, we are now issuing a RocksDB scan operation for every outbox read by the shuffle. This is highly inefficient and should rather use a tailing iterator similar to how the LocalLogletReadStream is implemented.

tillrohrmann avatar Jun 19 '24 11:06 tillrohrmann

From @tillrohrmann:

With the global total order log the shuffle will probably go away and improving it might not be worth it at this point.

Let's not pick this until it's clear that this optimisation is needed long-term.

pcholakov avatar Jul 31 '24 12:07 pcholakov