Alex Chi Z.

this week: continue investigation

Running the previous aux file pagebench does not see any perf regressions with 10k files `cargo run --bin pagebench aux-files`. So I guess this tenant experiencing slow basebackup is more...

for this tenant, no aux image file was generated since LSN 00000210492E07B8 (generation 000001fa).

By force running manual compaction, this can be resolved -- I can run this in staging while keep investigating locally. Seems that aux compaction is not triggered correctly in this...

Basebackup is unstuck for that project (temporarily), compute cannot start due to wal_level (compute team taking over to investigate) -- still need to investigate why image layers are not generated.

this week: investigate if we can improve read path by not tracking keys on sparse keyspace

another interesting observation is that the aux file layers are usually super small after L0->L1 compaction: ``` -rw-r--r-- 1 skyzh staff 224K Nov 4 15:46 030000000000000000000000000000000001-62000002011CDE14934B1DC19112DDCD798B__00000216783D4AC9-0000021678403A51-v1-000001fc -rw-r--r-- 1 skyzh staff...

we have two fixes: make read path faster, and make compaction more aggressive (while it shouldn't affect the amplification b/c the aux files are really small after making the updates...

The reason I didn't merge this patch is that I don't fully understand why RocksDB produces that sequence -- I think we got the first few lines correct, but looking...

after looking at it for a while, I assume it's a typo, though I suppose their example should be generated with some tools, but I couldn't figure out a reasonable...