dremio-oss
dremio-oss copied to clipboard
Make Query result limit value configurable instead of hardcoded 500 to reduce roundtrips to support also queries returning dozens of thousands rows with a few columns
With 500 hardcoded value its extremely unperformant having roundtrips to collect large amount of data. As for now Dremio doesnt allow to download results for queries posted via REST API, we have to iterate multiple times having limit 500 until the end. It causes an extra time and server workload. The PR is too increase the value, and also fixes the bug if user provides a negative limit.
I hate to totally 180 on my previous comments... I still think this should be configurable, but in my rapidly changing experience with the Dremio source code, this is the type of setting that Dremio would normally expose as a "Support Key", that you would provide through the Admin screen under Support.
Dremio.conf is a pretty rigid config file, with all settings being defined in advance, support keys seem a bit more flexible.
@patricker I moved that config value to optionManager