Reduce the Pebble compaction debt

[rjl493456442]

After exposing additional Pebble metrics, one notable observation is that Pebble shows around 8-9GB of compaction debt.

This indicates that there is approximately 8-9GB of data pending compaction, but compactions are not being scheduled.

Upon further investigation, lowering the L0CompactionThreshold can help force Pebble to schedule more compactions:

// The amount of L0 read-amplification necessary to trigger an L0 compaction.
L0CompactionThreshold int

This field sets the threshold for L0 read-amplification, which typically corresponds to the number of sublevels in Level 0. The default value is 4. Reducing it to 2 can significantly decrease compaction debt (to around 1GB) by triggering compactions earlier.

It has a few benefits, e.g., after launching the Geth, Pebble will forcibly compact the debt and block the program. With this change, the wait time can be very trivial.

However, a few things need to be measured before applying the change:

• What's the influence to the frontend read/write by scheduling compaction more offen

[nadtech-hub]

I could work on it as well