Eric Liang
Yeah, the other alternative I was thinking is to have two separate types of Block accessors entirely. On Wed, Jun 22, 2022, 5:39 PM Clark Zinzow ***@***.***> wrote: > @ericl...
I see. I think this is pointing to a larger code structure issue that our data format (ArrowBlock) should not also define execution methods (e.g., combine, etc.). What if we...
Hmm that's a good question, we should validate this for sure prior to merging.
Simple enough to run a trivial dataset workload with small blocks before/after this PR. Maybe like 10000 blocks range + map batches?
Some broken tests. Are there any blocking concerns to revisit the design? The microbench result seems expected / fine.
@jianoaix what's the status of this?
Ah, that makes sense. Either full sized or the status quo seems reasonable to me then!
I don't know the full history, but can't the cuda devices be hash identified instead of index identified? That would require a string and not an int.
Would +1 always strings, we can explain why in the doc.