restate
restate copied to clipboard
Update shuffle to use tailing iterator for performance improvements
With the fix for #1639, we are now issuing a RocksDB scan operation for every outbox read by the shuffle. This is highly inefficient and should rather use a tailing iterator similar to how the LocalLogletReadStream is implemented.
From @tillrohrmann:
With the global total order log the shuffle will probably go away and improving it might not be worth it at this point.
Let's not pick this until it's clear that this optimisation is needed long-term.