doris icon indicating copy to clipboard operation
doris copied to clipboard

[feature-wip](new-scan) add scanner scheduling framework

Open morningman opened this issue 3 years ago • 0 comments

Proposed changes

Issue Number: close #xxx

Problem summary

There are currently many types of ScanNodes in Doris. And most of the logic of these ScanNodes is the same, including:

  1. Runtime filter
  2. Predicate pushdown
  3. Scanner generation and scheduling

So I intend to unify the common logic of all ScanNodes. Different data sources only need to implement different Scanners for data access. So that the future optimization for scan can be applied to the scan of all data sources, while also reducing the code duplication.

This PR mainly adds three new class:

  1. VScanner All Scanners' parent class. The subclasses can inherit this class to implement specific data access methods.
  2. VScanNode The unified ScanNode, and is responsible for common logic including RuntimeFilter, predicate pushdown, Scanner generation and scheduling.
  3. ScannerScheduler Unified responsible for all Scanner scheduling tasks

TODO:

  1. Gradually migrate all existing ScanNodes to VScanNode.

Checklist(Required)

  1. Does it affect the original behavior:
    • [ ] Yes
    • [ ] No
    • [ ] I don't know
  2. Has unit tests been added:
    • [ ] Yes
    • [ ] No
    • [ ] No Need
  3. Has document been added or modified:
    • [ ] Yes
    • [ ] No
    • [ ] No Need
  4. Does it need to update dependencies:
    • [ ] Yes
    • [ ] No
  5. Are there any changes that cannot be rolled back:
    • [ ] Yes (If Yes, please explain WHY)
    • [ ] No

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

morningman avatar Aug 07 '22 16:08 morningman