paimon icon indicating copy to clipboard operation
paimon copied to clipboard

[core] Support statistic with time travel

Open xuzifu666 opened this issue 1 year ago • 5 comments

Purpose

Scenario:user query statistic table by snapshot-id and analyzed table multiple times. Such as user can do the analyzed at regular time and collect the statistic metric to monitor. Currently paimon query statistic system table with snapshot_id would error out due to PaimonAnalyzeTableColumnCommand##commitStatistics not keep the real snapshot id the same with statistic snapshot_id,the pr is aimed to fix it. before fix: 1727191085648.png

after fix: 1727191088377.png

Linked issue: close #xxx

Tests

AnalyzeTableTestBase "Paimon analyze: test statistic system table with predicate"

API and Format

Documentation

xuzifu666 avatar Sep 24 '24 15:09 xuzifu666

Good Job! But when I review your code, I found there is no doc for statistic system table. Could you add?

wwj6591812 avatar Sep 25 '24 01:09 wwj6591812

Good Job! But when I review your code, I found there is no doc for statistic system table. Could you add?

Addressed, Thanks for @wwj6591812 reivew, I would add a doc for it.

xuzifu666 avatar Sep 25 '24 01:09 xuzifu666

+1

wwj6591812 avatar Sep 25 '24 06:09 wwj6591812

traversal logic do a optimize, before: need from latest snapshot to the target analyzed snapshot, after: change from the nearest snapshot to the target analyzed snapshot. @JingsongLi 1727419450979.png

xuzifu666 avatar Sep 27 '24 06:09 xuzifu666

After discuss, change to query by real snapshot id. @JingsongLi

xuzifu666 avatar Sep 27 '24 08:09 xuzifu666