DistributedArrays.jl icon indicating copy to clipboard operation
DistributedArrays.jl copied to clipboard

Handling of DArray input to distributing functions

Open andreasnoack opened this issue 7 years ago • 4 comments

This is not what you want

julia> distribute(distribute(randn(4)))
4-element DistributedArrays.DArray{Float64,1,SubArray{Float64,1,DistributedArrays.DArray{Float64,1,Array{Float64,1}},Tuple{UnitRange{Int64}},false}}:
  1.27283
 -0.642431
 -0.585599
 -1.96976

The most general solutino might be to change verified_destination_serializer. Alternatively, we'll have to add special DArray methods for each method defined for AbstractArray. @amitmurthy thoughts?

andreasnoack avatar Oct 03 '16 19:10 andreasnoack

Defining this should suffice, right?

distribute(d::DArray) = DArray(I->convert(Array, d[I...]), d)

amitmurthy avatar Oct 04 '16 08:10 amitmurthy

It might be the right solution but we might also want to use the verified_destination_serializer in other places. We are using it in scale! and if the scaling vector is already distributed then we have the same problem. This is the reason for the last sentence in the previous comment. It might be that we should just add methods specifically for DArray for each function defined for AbstractArray but it would be nice if we didn't have to.

andreasnoack avatar Oct 04 '16 13:10 andreasnoack

I modified verified_destination_serializer in a local branch, but I ended up needing changes in distribute too. I feel it is cleaner and better to add methods specific to DArray.

amitmurthy avatar Oct 05 '16 16:10 amitmurthy

Fine. Thanks for looking into it.

andreasnoack avatar Oct 05 '16 16:10 andreasnoack