refactor(s3stream/object-wal): complete appends sequentially

Open Chillax-0v0 opened this issue 8 months ago • 1 comments

resolve #2432

Performance Test

Before

2025-04-18 07:42:14 - INFO Summary | Prod rate   1600.46 msg/s /  79.71 MiB/s | Prod total   0.48 M msg /  23.36 GiB /   0.00 K err | Cons rate   1600.44 msg/s /  79.71 MiB/s | Cons total   0.48 M msg /  23.36 GiB | Prod Latency (ms) avg:   267.319 - 50%:   233.070 - 75%:   276.479 - 90%:   416.589 - 95%:   545.547 - 99%:   759.195 - 99.9%:   972.191 - 99.99%:  1230.271 - Max:  1250.511 | E2E Latency (ms) avg:   272.780 - 50%:   237.312 - 75%:   280.909 - 90%:   436.249 - 95%:   547.739 - 99%:   778.615 - 99.9%:   981.371 - 99.99%:  1238.991 - Max:  1381.615

After

2025-04-18 08:11:08 - INFO Summary | Prod rate   1601.42 msg/s /  79.76 MiB/s | Prod total   0.48 M msg /  23.40 GiB /   0.00 K err | Cons rate   1600.99 msg/s /  79.74 MiB/s | Cons total   0.48 M msg /  23.40 GiB | Prod Latency (ms) avg:   285.990 - 50%:   243.027 - 75%:   301.301 - 90%:   484.555 - 95%:   598.403 - 99%:   784.067 - 99.9%:   912.219 - 99.99%:  1027.723 - Max:  1064.903 | E2E Latency (ms) avg:   302.057 - 50%:   250.507 - 75%:   324.333 - 90%:   521.625 - 95%:   628.719 - 99%:   831.655 - 99.9%:  1001.287 - 99.99%:  1099.751 - Max:  1202.439

Apr 17 '25 10:04 Chillax-0v0

Should add unit tests to cover this refactor, and consider creating subsequent issues and PRs to simplify the upper-level logic.

Apr 24 '25 09:04 daniel-y