qpmodel
qpmodel copied to clipboard
use execution feedback to improve CE / re-optimization / AQP
The theory is here: https://arxiv.org/abs/1711.08330 and an implementation with PostgreSQL is here: https://github.com/postgrespro/aqo
A related topic is re-optimization (https://arxiv.org/pdf/1902.08291.pdf), where when executor detects discrepancy between optimizer's estimation and actual data, query can kick out reoptimization to fully redo or partially redo the query. It is also called adaptive query execution (in spark's term) where 3 adjustments made:
- shuffle adjust
- join method
- join skew handling
Note that spark's staged execution is easier for this mode because it can clearly cut done|not done parts of the query but mppdb stylestreaming based queries needs more designs (similar to staged query restart idea).