databend
databend copied to clipboard
try avoid compactor for copy into/streaming load with row-based format
for copy into/streaming load with row-based format, the best place to cut rows to blocks is block builder.
compactor can be avoided (not need to remove this processor, only not do any real compact is enough)
need to:
- get the 3 related tables options (but fuse table only) of the writing table, and let the deserialize processor of StageTable know it
- able to estimate the size (not rows) of a building block. (but a cost to check it after each row append, check it in an adaptive interval: estimate row size, )
I will start with use DEFAULT_ROW_PER_BLOCK * 0.8 ( try not to exceed size limit)
another choice is to produce small block from stageTable. guess merging tables is faster than splitting it?
@zhyass @dantengsky