Improving Query Preemption on Brokers/Servers

[jasperjiaguo]

Currently, the broker/server pre-empt a query at certain points during scheduling/execution based on timeOut. However, this is not sufficient for a couple of use-cases. Few examples:

• When a user wants to initiate some ad-hoc queries for data analysis on a large production table but doesn't specify a tight data range.
• When unexpected slow queries are saturating CPU/memory on broker and servers.
• When a large report generation query (especially with large groupby+orderby/select*/other compute intensive functions) is initiated.

Any of these cases can cause a broker/server to be very slow or bring down a host, cause OOM etc. Our pre-emption logic can be extended to both brokers and servers to pre-empt queries based on couple of factors such as:

- CPU Time Broker currently has logic to return queries that have reached timeout. However, this can be improved at the broker/server. We can potentially account for the total CPU workload on all worker threads on a server.

- Memory consumption - to avoid out-of-memory errors We want to estimate the heap memory usage and pre-empt based on a threshold.

We would like to find the correct instrument of measuring these factors and improve preemption with least overhead. We have seen few cases in production where customers have run bad queries that result in pinot broker or server crashing with OOM.

Other side of query resiliency being worked in https://github.com/apache/pinot/issues/8618

[Jackie-Jiang]

cc @atris since this is related to the query resiliency and cost estimation

[jasperjiaguo]

cc @atris since this is related to the query resiliency and cost estimation

Listed are some potential approaches we are evaluating/benchmarking. It's just a sketch pad; the design is not finalized. @vvivekiyer and I are still discussing. Please feel free to add any ideas/approaches. https://docs.google.com/document/d/1O_YKoDpyWjdxd55gE6zAuOjN8_wh0nNBAn9ocIGq25E/edit?usp=sharing

[siddharthteotia]

Update -- @jasperjiaguo is working on writing the design doc. Will share here once we have the first draft.

[siddharthteotia]

FYI @lakshmanan-v

[jasperjiaguo]

Update -- we would like to share the design doc here https://docs.google.com/document/d/1Z9DYAfKznHQI9Wn8BjTWZYTcNRVGiPP0B8aEP3w_1jQ/

Please feel free to review/comment

[siddharthteotia]

@vvivekiyer @jasperjiaguo and I met with @Jackie-Jiang and @walterddr to review the design doc and some code walkthrough.

Overall looks good. One main suggestion received was to unify / encapsulate the different kinds of info being held in different places (e.g Trace related info, the stats being introduced in this PR etc) into some form of ExecutionContext. Will be making those changes and send out the PR / first PR in a week or so.

[siddharthteotia]

Implementation for runtime query killing on server has been merged as part of https://github.com/apache/pinot/pull/9727

@jasperjiaguo is working to extend the support on broker as well. More or less similar design shared and reviewed earlier here but just extending it to work with broker execution code and thread model.

Here are the user docs to leverage the feature on server - https://docs.pinot.apache.org/operators/operating-pinot/oom-protection-using-automatic-query-killing

We did testing on actual production use case where expensive queries in the past have caused OOM. With the change enabled, we can see culprit queries getting killed live and server is protected from OOM.

[siddharthteotia]

PR should be out in a week or two for broker query killing support. @jasperjiaguo actively working on it.

[siddharthteotia]

PR is out for broker query killing support

[siddharthteotia]

Support for broker query killing has been merged

[siddharthteotia]

@jasperjiaguo - please make sure to add user docs asap.