doris
doris copied to clipboard
[feature-wip](new-scan) add scanner scheduling framework
Proposed changes
Issue Number: close #xxx
Problem summary
There are currently many types of ScanNodes in Doris. And most of the logic of these ScanNodes is the same, including:
- Runtime filter
- Predicate pushdown
- Scanner generation and scheduling
So I intend to unify the common logic of all ScanNodes. Different data sources only need to implement different Scanners for data access. So that the future optimization for scan can be applied to the scan of all data sources, while also reducing the code duplication.
This PR mainly adds three new class:
- VScanner All Scanners' parent class. The subclasses can inherit this class to implement specific data access methods.
- VScanNode The unified ScanNode, and is responsible for common logic including RuntimeFilter, predicate pushdown, Scanner generation and scheduling.
- ScannerScheduler Unified responsible for all Scanner scheduling tasks
TODO:
- Gradually migrate all existing ScanNodes to VScanNode.
Checklist(Required)
- Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
- Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
- Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
- Does it need to update dependencies:
- [ ] Yes
- [ ] No
- Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...