Renjie Liu

Improve distributed query engine according to this design: https://singularity-data.quip.com/DMRPAbITntNc/Distributed-Query-Processing-Engine-Design Query manager(frontend): - [x] #2422 - [x] #2423 - [x] #3830 - [ ] #3831 - [x] #4051 - [x] #5028...

batch: Execute root fragment task in frontend.

4

Currently the root fragment of mpp query is executed in compute node. It would be better to move it to frontend to reduce schedule and pull data latency.

batch: Setup benchmark for executors.

5

Help to understand the performance with our implementation. - [x] #3530 - [x] #3531 - [ ] #3532 - [ ] #3535 - [ ] #3536 - [ ] #3537...

batch: Add metrics for tasks.

1

Add task level metrics, including incoming number of rows and data size of each exchange source, outgoing number of rows and data size of each output.

Support data exporter. For implementation, we can utilize the Serialize of `QueryStage`.

Upgrade `rand` version in v2.12.0 to avoid security issue.

2

Hi, please update `rand` version in v2.12.0 to avoid security issue. I've created a patch but don't know which brach should I create PR against.

bug: Failed to load gradle project.

2

### Describe the bug Scala metals failed to load gradle project. 1. git clone [email protected]:apache/iceberg.git 2. Open it with vscode metals. 3. It failed to load project with following error:...

The content of each operator should be aligned with datafusion plan nodes, except inputs.

Currently, the set of rules can be enumerated. As an optimizer framework, the rule set should be extensible, e.g. the optimizer should accept some configurations and rule sets as `Vec`