restatedev
Partition shuffle should use background appenders
When a single partition is producing a lot of outbound calls (eg, in an aggressive fanout case), the shuffler quickly becomes a bottleneck, as it currently is unbatched, so it sends each shuffled message one at a time, and waits for it to be accepted by bifrost before moving on to the next. When append durations are a little high, this can really impact throughput.
If shuffle used a background appender (per destination log id), removing items from the outbox once they are confirmed appended, we might see significant throughput improvements as the items could be batched and also handled concurrently across logs. In principle, there could be very large batches when user code has emitted large numbers of outbound calls.
