Optimize case of high join amplification

[kwannoel]

In streaming workload, high join amplification can lead to barrier latency spike.

So far our approaches to mitigate these have been more indirect, e.g. decoupling sink, adaptive rate limit. It can isolate the slowness to the specific stream job.

I think we should also explore optimizing join itself.

• [ ] Come up with a high cardinality join scenario, with few columns.
• [ ] Come up with a high cardinality join scenario, with many columns.
• [ ] Observe the flamegraph to see where it can be optimized further.
• [ ] Investigate other systems to see what optimizations can be made in similar situations, e.g. cross join.