JuliaParallel
array constructors should have automatic mode
Currently, a user has to specify the block partition sizes. Ideally, there should be an automatic choice, perhaps by distributing on the last dimension, or some other well defined method.
@ViralBShah it's been a few years, but is there some example code that illustrates this, as well as a rough draft of what you'd like the API to look like?
I was thinking something automatic would automatically do a 1d block distribution of roughly equal sizes. Maybe pick the same defaults and API as DistributedArrays.jl.
Some other ranting:
Dagger.jl's scheduler should probably be separated from the array interface, which might live in DaggerArrays.jl or so.
BTW, are you thinking that this is the right place to improve distributed arrays? One possibility is to simply improve DistributedArrays.jl. This is a huge design space, but in-memory distributed arrays don't seem to need a dynamic scheduler and its overhead. If we want these things to be out-of-core, then yes all this is worthwhile.
I agree that the array interface could do to be separate from the scheduler in this package, since it's clearly a decent chunk of code that the scheduler is generally agnostic to. But that's also a lot of work that I'm not comfortable with handling right now, so someone else would have to take up that torch.
Regarding DistributedArrays, I think that it would make sense to improve that package instead of trying to roll our own here just for this one method of scheduling. However, we'll also want the ability for distributed array operations to be broken into chunks by the scheduler if it knows enough about the performance characteristics of said operation w.r.t the characteristics (latency, throughput, IPC, etc.) of the nodes it controls (something I plan to implement in the near-ish future).
Back to the original topic: if you can show me an example of a verbose, manual way to distribute blocks that you think could become automatic, please post it here and I'll try to wrap my head around the array interface enough to implement your request :smile:
The issue is now unfortunately too old for me to remember the exact API I was using, and this would have come up when @shashi and I were looking at things together.