[TASK] Investigate data plane optimization potential

[joshimoo]

What's the task? Please describe

Take a look at the engine architecture and consider optimization potentials: -> #3176 read caching

Profile the internal data flow (tgt -> engine -> replica -> disk) To identify optimization potential surrounding latency and througput -> memory overhead reduction -> lower gc pressure

[innobead]

Need to pay more focus on performance tuning, so move to p1.

[joshimoo]

I did some investigations on the data plane, and I think we can get 5-10% as well overall reduced memory usage withhout big changes to the architecture :)

I will write up my findings and the necessary changes/improvements next week.

[joshimoo]

Performance Benchmarks between lh v1.2.x and v1.3 https://github.com/longhorn/longhorn/issues/4002

Single local replica via unix socket latency decrease benchmark https://github.com/longhorn/longhorn/issues/3957

[derekbit]

Replace the gcc memory allocation with efficient one https://github.com/longhorn/longhorn/issues/4136

[innobead]

Replace the gcc memory allocation with efficient one #4136

note: tcmalloc has been reverted in 1.3.3 and 1.4.0.

[innobead]

Let's use https://github.com/longhorn/longhorn/issues/6600 instad.