Andy Grove
Andy Grove
I plan on working on this since I think it will solve some problems I am running into when working with expressions and schemas.
I spent some time on this and I am no longer sure that this change makes sense. We have a lot of optimizer rules that recurse through expression trees rewriting...
I am investigating this
Here is some debug output from `ordered distribution and sort with same exprs: append` when running with Comet enabled: Columnar shuffle wrote the following shuffle bytes (using the default lz4...
In Comet, we write the schema with each record batch. That likely explains the difference, especially with these tests which are writing 10 rows.
> In Comet, we write the schema with each record batch. That likely explains the difference, especially with these tests which are writing 10 rows. @kazuyukitanimura @comphead fyi
> > In Comet, we write the schema with each record batch. That likely explains the difference, especially with these tests which are writing 10 rows. > > [@kazuyukitanimura](https://github.com/kazuyukitanimura) [@comphead](https://github.com/comphead)...
I am going to start working on this.
The "unified" approach certainly seems much safer and simpler. I have been benchmarking locally with `spark.memory.offHeap.enabled=true` and the benchmarking guide does not reflect this. I am going to start out...
> Thank you so much for declaring the recommended setup! It's given us a great direction, especially towards enhancing the memory manager in "unified" mode. Currently, I think the absence...