quackpipe

quackpipe copied to clipboard

We need to make sure any metadata obsolete/orphaned parquet files are consistently removed from storage once the compaction cycle has completed or as part of recovery/restarts to garbage collect any compaction leftovers from previous runs as well.

A CLI mode to request compaction of a specific range to a specific level could be beneficial for recovery scenarios