DArray roadmap

[jakebolewski]

object identity and equality

• [x] copy
• [x] deepcopy (aliases)
• [ ] hash (works but through the AbstractArray fallback)
• [x] identity
• [x] isbits
• [ ] isequal (works but through the AbstractArray fallback)
• [x] isimmutable
• [ ] lexless (works but through the AbstractArray fallback. It just requires lexcmp)
• [ ] lexcmp (works but through the AbstractArray fallback)
• [x] object_id
• [x] sizeof

collections

• [x] all!
• [x] all
• [x] any!
• [x] any
• ~~collect~~
• [ ] contains (works but through the AbstractArray fallback)
• [x] count
• [ ] deleteat!
• [x] eltype
• [ ] empty!
• [x] endof
• [ ] filter!
• [ ] filter
• [ ] foldl (works but through the AbstractArray fallback)
• [ ] foldr (works but through the AbstractArray fallback)
• [x] get
• [ ] get!
• [x] getindex
• [ ] in (works but through the AbstractArray fallback)
• [ ] intersect!
• [ ] intersect (works but through the AbstractArray fallback)
• [x] isempty
• [ ] issubset (works but through the AbstractArray fallback)
• [x] length
• [x] map!
• [x] map
• [ ] mapfoldl (works but through the AbstractArray fallback)
• [ ] mapfoldr (works but through the AbstractArray fallback)
• [x] mapreduce
• [x] mapreducedim
• [x] reduce
• [ ] setdiff (works but through the AbstractArray fallback)
• [ ] setindex!
• [x] similar
• ~~sizehint!~~ (Probably doesn't make sense)
• [ ] splice!
• [ ] symdiff (works but through the AbstractArray fallback)
• [ ] union (works but through the AbstractArray fallback)
• [ ] unique (works but through the AbstractArray fallback)

iteration (works but through the AbstractArray fallback. This requires some care to work well)

• [ ] done
• [ ] enumerate
• [ ] next
• [ ] start
• [ ] zip
• [ ] rest
• [ ] countfrom
• [ ] take
• [ ] drop
• [ ] cycle
• [ ] repeated

arrays

• [ ] broadcast!
• [x] broadcast
• [ ] cat
• [x] checkbounds
• [ ] circshift
• [ ] conj!
• [x] copy!
• [ ] cummax
• [ ] cummin
• [ ] cumprod
• [ ] cumprod!
• [ ] cumsum
• [ ] cumsum!
• [ ] cumsum_kbn
• [x] eachindex
• [ ] extrema (works but through the AbstractArray fallback)
• [x] fill!
• [ ] find (works but through the AbstractArray fallback)
• [ ] findfirst (works but through the AbstractArray fallback)
• [ ] findlast (works but through the AbstractArray fallback)
• [ ] findin (works but through the AbstractArray fallback)
• [ ] findmax (works but through the AbstractArray fallback)
• [ ] findmin (works but through the AbstractArray fallback)
• [ ] findn (works but through the AbstractArray fallback)
• [ ] findnext (works but through the AbstractArray fallback)
• [ ] findprev (works but through the AbstractArray fallback)
• [ ] findnz (works but through the AbstractArray fallback)
• [ ] first (works but through the AbstractArray fallback)
• [ ] flipdim
• ~~gradient~~ (will probably go away in base)
• [ ] hcat
• [ ] hvcat
• [x] ind2sub
• [ ] indexin (works but through the AbstractArray fallback)
• [ ] indmax (works but through the AbstractArray fallback)
• [ ] indmin (works but through the AbstractArray fallback)
• [ ] invperm
• [ ] ipermute!
• [ ] ipermutedims
• [ ] isperm (works but through the AbstractArray fallback)
• [ ] issorted (works but through the AbstractArray fallback)
• [ ] last (works but through the AbstractArray fallback)
• [ ] linspace (dlinspace?)
• [ ] logspace (dlogspace?)
• [x] mapslices
• [ ] max (works but through the AbstractArray fallback)
• [x] maxabs
• [ ] maxabs!
• [ ] maximum!
• [x] maximum
• [ ] min (works but through the AbstractArray fallback)
• [x] minabs
• [ ] minabs!
• [ ] minimum!
• [x] minimum
• [ ] minmax
• [x] ndims
• [x] nnz
• [ ] nonzeros
• [x] countnz
• [x] ones
• [x] parent
• [x] parentindexes
• [ ] permute!
• [ ] permutedims
• [ ] permutedims!
• [ ] prod!
• [x] prod
• [x] promote_shape
• [ ] randcycle (drandcycle)
• [ ] randperm (drandperm)
• [ ] randsubseq! (works but through the AbstractArray fallback)
• [ ] randsubseq (works but through the AbstractArray fallback)
• [x] reducedim
• [ ] repmat
• [x] reshape
• [ ] reverse!
• [ ] reverse
• [ ] rot180
• [ ] rotl90
• [ ] rotr90
• [ ] searchsorted (works but through the AbstractArray fallback)
• [ ] searchsortedfirst (works but through the AbstractArray fallback)
• [ ] searchsortedlast (works but through the AbstractArray fallback)
• [ ] select!
• [ ] select
• [ ] shuffle
• [ ] shuffle!
• [x] size
• [x] view
• [ ] slicedim
• [ ] sort!
• [x] sort
• [ ] sortcols
• [ ] sortperm (works but through the AbstractArray fallback)
• [ ] sortperm!
• [ ] sortrows
• [ ] squeeze
• [ ] step
• [ ] stride
• [ ] strides
• [ ] sub
• [ ] sub2ind
• [ ] sum!
• [x] sum
• [ ] sumabs!
• [x] sumabs
• [ ] sumabs2!
• [x] sumabs2
• [ ] sum_kbn
• [ ] vcat
• [ ] vec
• [x] zeros

linear algebra

• [ ] bkfact!
• [ ] bkfact
• [ ] blas_set_num_threads
• [ ] blkdiag
• [ ] chol
• [ ] cholfact!
• [ ] cholfact
• [ ] cond
• [ ] condskeel
• [ ] cross
• [ ] ctranspose!
• [x] ctranspose
• [ ] det
• [ ] diag
• [ ] diagind
• [ ] diagm
• [ ] diff
• [ ] dot
• [ ] eig
• [ ] eigfact!
• [ ] eigfact
• [ ] eigmax
• [ ] eigmin
• [ ] eigs
• [ ] eigvals
• [ ] eigvals!
• [ ] eigvecs
• [ ] expm
• [ ] eye
• [ ] factorize
• [ ] givens
• [ ] hessfact!
• [ ] hessfact
• [ ] ishermitian
• [ ] isposdef!
• [ ] isposdef
• [ ] issym
• [ ] istril
• [ ] istriu
• [ ] kron
• [ ] ldltfact
• [ ] ldltfact!
• [ ] linreg
• [ ] logdet
• [ ] lu
• [ ] lufact!
• [ ] lufact
• [ ] lyap
• [ ] norm
• [ ] nullspace
• [ ] ordschur!
• [ ] ordschur
• [ ] peakflops
• [ ] pinv
• [ ] qr
• [ ] qrfact!
• [ ] qrfact
• [ ] rank
• [x] scale!
• [ ] scale
• [ ] schur
• [ ] schurfact!
• [ ] schurfact
• [ ] sqrtm
• [ ] svd
• [ ] svdfact!
• [ ] svdfact
• [ ] svds
• [ ] svdvals!
• [ ] svdvals
• [ ] sylvester
• [ ] trace
• [ ] transpose!
• [x] transpose
• [ ] tril!
• [ ] tril
• [ ] triu!
• [ ] triu
• [ ] vecnorm

sparse

• [ ] etree
• [ ] full
• [ ] issparse
• [ ] sparse
• [ ] sparsevec
• [ ] spdiagm
• [ ] speye
• [ ] spones
• [ ] sprand
• [ ] sprandbool
• [ ] sprandn
• [ ] spzeros
• [ ] symperm
• [ ] rowvals
• [ ] nzrange

Operators

• [x] !
• [x] !=
• [x] ≠
• [x] !==
• [x] ≡
• [x] ≢
• [x] *
• [x] +
• [x] -
• [ ] /
• [x] //
• [x] ==
• [ ] \
• [ ] ^
• [ ] A_ldiv_B!
• [ ] A_ldiv_Bc
• [ ] A_ldiv_Bt
• [ ] A_mul_B!
• [ ] A_mul_Bc
• [ ] A_mul_Bc!
• [ ] A_mul_Bt
• [ ] A_mul_Bt!
• [ ] A_rdiv_Bc
• [ ] A_rdiv_Bt
• [ ] Ac_ldiv_B
• [ ] Ac_ldiv_Bc
• [x] Ac_mul_B
• [x] Ac_mul_B!
• [ ] Ac_mul_Bc
• [ ] Ac_mul_Bc!
• [ ] Ac_rdiv_B
• [ ] Ac_rdiv_Bc
• [ ] At_ldiv_B
• [ ] At_ldiv_Bt
• [ ] At_mul_B
• [ ] At_mul_B!
• [ ] At_mul_Bt
• [ ] At_mul_Bt!
• [ ] At_rdiv_B
• [ ] At_rdiv_Bt

random numbers

• [ ] rand!
• [x] rand
• [ ] randn!
• [x] randn
• [ ] randexp!
• [ ] randexp
• [ ] srand
• [ ] bitrand

statistics

• [ ] cor
• [ ] cov
• [x] mean!
• [x] mean
• [ ] median!
• [x] median
• [ ] middle
• [ ] midpoints
• [ ] quantile!
• [ ] quantile
• [x] std
• [x] stdm
• [x] var
• [x] varm

signal processing

• [ ] bfft!
• [ ] bfft
• [ ] brfft
• [ ] conv
• [ ] conv2
• [ ] dct!
• [ ] dct
• [ ] deconv
• [ ] fft!
• [ ] fft
• [ ] fftshift
• [ ] filt
• [ ] filt!
• [ ] idct!
• [ ] idct
• [ ] ifft!
• [ ] ifft
• [ ] ifftshift
• [ ] irfft
• [ ] plan_bfft!
• [ ] plan_bfft
• [ ] plan_brfft
• [ ] plan_dct!
• [ ] plan_dct
• [ ] plan_fft!
• [ ] plan_fft
• [ ] plan_idct!
• [ ] plan_idct
• [ ] plan_ifft!
• [ ] plan_ifft
• [ ] plan_irfft
• [ ] plan_rfft
• [ ] rfft
• [ ] xcorr

Update 30 September 2016 by @andreasnoack: I've removed all the scalar math and special functions. With the . syntax + broadcast(x::DArray) = map(x) they should work.

[jiahao]

For a first pass we can skip most of the vectorized functions - i.e. all the specfuns and most of the scalar mathematical functions.

[ViralBShah]

Or implement them with macros and conquer much of this list. :-)

I love this list. I am sure @alanedelman loves it too.

[ViralBShah]

621 functions. 16 done. Only 605 to go. :-)

Cc: @amitmurthy @tanmaykm @alanedelman in case you guys haven't seen this.

[jakebolewski]

Some of these will be hard / impossible to do in parallel. My favorite is == if we want to support the same semantics as AbstractArrays now in base.

[ViralBShah]

Yes, of course!

What is the exact issue with == ? Perhaps should be a separate issue.

[jiahao]

@ViralBShah I agree == (with hash) should be their own issue. The challenge is how to hash DArrays in parallel so that they can test equal under == regardless of distribution. The only way we can guarantee that now for A==B is to redistribute all the data onto a single process and do the hash locally for A and B.

[jakebolewski]

@tanmaykm, @amitmurthy it would be great if you could check off functions as they are added. I believe anyone that is an owner of the repo can edit the check list.

[alanedelman]

don't we have svd and a few other scalapack functions now?

[ViralBShah]

We should probably separate out linear algebra related function checklist into scalapack.jl or elemental.jl, and have this issue track only DArrays.

[alanedelman]

but i'd like this to work with darrays :-)

On Thu, Apr 23, 2015 at 8:27 AM, Viral B. Shah [email protected] wrote:

We should probably separate out linear algebra related function checklist into scalapack.jl or elemental.jl, and have this issue track only DArrays.

— Reply to this email directly or view it on GitHub https://github.com/JuliaParallel/DistributedArrays.jl/issues/4#issuecomment-95568816 .

[andreasnoack]

It will, but the dependency order right now is that ScaLAPACK.jl depends on DistributedArrays.jl and, therefore, the DArray+ScaLAPACK functionality is in ScaLAPACK.jl.

[andreasnoack]

@alanedelman Just to avoid misunderstanding here. You can do

julia> using MPI

julia> man = MPIManager(np = 4);

julia> addprocs(man);

julia> using ScaLAPACK
Warning: Method definition defaultdistTuple{Int64,Int64} in module DistributedArrays at /Users/andreasnoack/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:147 overwritten in module ScaLAPACK at /Users/andreasnoack/.julia/v0.4/ScaLAPACK/src/ScaLAPACK.jl:13.
    From worker 4:  Warning: Method definition defaultdistTuple{Int64,Int64} in module DistributedArrays at /Users/andreasnoack/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:147 overwritten in module ScaLAPACK at /Users/andreasnoack/.julia/v0.4/ScaLAPACK/src/ScaLAPACK.jl:13.
    From worker 3:  Warning: Method definition defaultdistTuple{Int64,Int64} in module DistributedArrays at /Users/andreasnoack/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:147 overwritten in module ScaLAPACK at /Users/andreasnoack/.julia/v0.4/ScaLAPACK/src/ScaLAPACK.jl:13.
    From worker 2:  Warning: Method definition defaultdistTuple{Int64,Int64} in module DistributedArrays at /Users/andreasnoack/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:147 overwritten in module ScaLAPACK at /Users/andreasnoack/.julia/v0.4/ScaLAPACK/src/ScaLAPACK.jl:13.
    From worker 5:  Warning: Method definition defaultdistTuple{Int64,Int64} in module DistributedArrays at /Users/andreasnoack/.julia/v0.4/DistributedArrays/src/DistributedArrays.jl:147 overwritten in module ScaLAPACK at /Users/andreasnoack/.julia/v0.4/ScaLAPACK/src/ScaLAPACK.jl:13.

julia> @everywhere using DistributedArrays

julia> A = drandn((1000, 1000), man);

julia> B = convert(Array, A);

julia> @time valsA = svdvals(A);
elapsed time: 1.382340321 seconds (37 MB allocated, 1.53% gc time in 1 pauses with 0 full sweep)

julia> @time valsA = svdvals(A);
elapsed time: 0.323082764 seconds (1 MB allocated)

julia> @time valsB = svdvals(B);
elapsed time: 1.0682854 seconds (11 MB allocated, 0.48% gc time in 1 pauses with 0 full sweep)

julia> @time valsB = svdvals(B);
elapsed time: 0.998921006 seconds (8 MB allocated)```
The warnings will go away eventually.

[alanedelman]

I was kind of sad that now that we are about to have the "dot" (Pronounced broadcast but I don't like that name, I prefer pointwise or just point) why we don't have these functions after all these years.

Why can't we have all the operators and all the dot operators easily?

[andreasnoack]

We should have all the dot stuff now but it only works on Julia 0.6 and master. Somebody just needs to go through the list and check. Some of the items are also deprecated now.

[andreasnoack]

@alanedelman I just checked and all the dot stuff works now for DArrays. However, they are not independent functions anymore but just syntax so I've removed them from the list. The important detail is that broadcast works.

[alanedelman]

where are we on cumsum, cumprod, and accumulate ?

[andreasnoack]

Still some work to be done there.

[alanedelman]

would love to bring this issue back to life

[montyvesselinov]

me too

[ViralBShah]

@christopher-dG Just bringing this list to your attention. Maybe we should discuss this today?

[alanedelman]

+1