seafowl icon indicating copy to clipboard operation
seafowl copied to clipboard

Expose some DataFusion runtime execution settings in the config

Open mildbyte opened this issue 2 years ago • 0 comments

Done in https://github.com/splitgraph/seafowl/pull/71: ~~- MemoryManagerConfig: max runtime memory usage for plan execution (rough, since it doesn't track basic process data structures): https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/execution/memory_manager.rs#L35-L55 / https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/execution/runtime_env.rs#L141-L145~~ ~~- DiskManagerConfig: using the OS temp directory / other path to spill data to disk during execution: https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/execution/disk_manager.rs#L50-L65 / https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/execution/runtime_env.rs#L135-L139~~

Misc:

  • Planner/executor config options from https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/config.rs and https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/execution/context.rs#L1038-L1060

mildbyte avatar Aug 12 '22 19:08 mildbyte