apache
Add default limit to multi-stage query engine
Pinot is not able to stream data from brokers to servers. This means that queries that return a large amount of data need keep at least copy in the broker (and depending on the case, another in some server). This may end up producing OOM errors in Pinot components.
On of the protections included in single-query engine is to add a default limit to queries that not include it. For example, a query like select * from Table is transformed into select * from Table limit 10.
This is an unexpected behavior and that is why Multi-stage query does not apply this technique. But there have been some discussion to apply this in multi-stage as well.
Ideally, we should have the ability to turn this behavior off so users can choose to follow standard SQL semantics (no default limit) or this mode where limits are added.
Notice that this is not enough to guarantee there will be no OOM errors.
