improve forward-a-chunk loop to support low tail-latency

[vigith]

Summary

Today, we have to fix constraints

• [ ] only an element can be processed at a time
• [ ] we create a map-stream (gRPC) for every batch (meaning streaming inside unary)
• [ ] our batch is a blocking batch and will not proceed if we have a slow message
• [ ] Read, Execute, Forward, Ack is an ordered (sync) workflow for a batch, we need to make it out-of-order non-blocking
• [ ] move away from blocking batch to bounded concurrency (X in a batch to always have X in the backlog)

We need to remove these constraints without affecting the correctness or completeness property.

Use Cases

• Better throughput

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.