Chunk ingestion performance regression because of compaction logic

[rgolovanov]

Describe the bug The latency in the Viewer is rapidly growing when dozens of scalar values are logged. The performance has degraded between 0.17.0 and 0.19.0 (or maybe even 0.18.1)

To Reproduce Steps to reproduce the behavior:

1. pip install rerun-sdk==0.17.0
2. python rerun_latency_test.py (see code below)
3. Check latency in performance metrics in Viewer
4. Repeat steps (1) - (3) for version 0.19.0 and see how quickly the latency goes up and above a few seconds.

Expected behavior The latency stays within 10-15ms range as it was in earlier versions.

Screenshots

Test script rerun_latency_test.py

probably, numbers can be different depending on the machine, but in my case the problem is visible when I log 10x5x3=150 scalar values.

import rerun as rr
import time
import random
import threading
import itertools

def log_scalar_values():
    rr.init(f"TestLatency", spawn=True, recording_id="default")
    while not stop_event.is_set():
        for i, j, k in itertools.product(range(0, 10), range(0, 5), range(0, 3)):
            scalar_value = random.uniform(0, 1)  # Generate a random scalar value
            rr.log(f"AAA/{i}/{j}/{k}/value", rr.Scalar(scalar_value))
            print(f"Log {scalar_value}")
        time.sleep(0.001)  # Sleep for 10ms to achieve approximately 100Hz


stop_event = threading.Event()

logging_thread = threading.Thread(target=log_scalar_values)
logging_thread.start()

input("Press Enter to stop logging...\n")
stop_event.set()
logging_thread.join()
print("Logging stopped.")

Desktop (please complete the following information):

• OS: Windows 11

Rerun version Detected in 0.19.0 but likely appeared in 0.18.0 or 0.18.1

[codspeed-hq[bot]]

CodSpeed Performance Report

Merging #4949 will degrade performances by 41.57%

_{Comparing mieslep:phil/effective-component-values (ffd3d4b) with main (ba31d43)}

Summary

⚡ 1 improvements
❌ 3 regressions
✅ 11 untouched benchmarks

:warning: Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	mieslep:phil/effective-component-values	Change
❌	test_setup_llm_caching	1.2 ms	2 ms	-41.57%
❌	test_successful_run_with_input_type_any	219.4 ms	347.1 ms	-36.8%
❌	test_successful_run_with_input_type_text	158.6 ms	250.4 ms	-36.65%
⚡	test_successful_run_with_output_type_any	245.5 ms	153.5 ms	+59.95%

[mieslep]

@ogabrielluiz I don't know what's going on with autofix.ci which has added 1b5c333 - this youtube transcript is not part of this PR, but I can't figure out how to keep it out :/