Alex Chi Z.
Alex Chi Z.
Need to investigate the repartition code path. I would prefer doing a hard split at different relations, and inside the keyspace of each relation, simply split on the target size...
I don't see big problem with the image layer size for now. We could reopen this later.
Yep I’ll figure it out in another patch.
Yes I was worried about perf given the few extra bytes added with the requests; I'll run a bench and see if everything works fine :)
I can see some regressions (the last data point was this patch): I'll find some other ways to continue the investigation :(
> What benefit do we get if we have this as a runtime config? If it still OOMs we can adjust to a lower limit... > Could we do this...
seems like there are quite a lot test cases asserting on the compaction result layer size/num, probably we should do `threshold*checkpoint*2` to avoid hitting those issues
PTAL, Thanks! @breeswish @TennyZhuang
I'll turn this PR into draft as more time is required to tackle the underlying issue.
please ping us for review once CI errors are fixed :) thank you