dash icon indicating copy to clipboard operation
dash copied to clipboard

Published 20 hours ago • verified publisher icon dash-project

Change distribution after create Matrix

Open Goon83 opened this issue 5 years ago • 6 comments

Hi All, Another question regarding the function availability of DASH.

What will happen if I change the distribution (block size, halo layer) of a existing Matrix ? Will the DASH support re-distribution the data following new parameters?

Bests, Bin

Goon83 avatar Mar 02 '20 22:03 Goon83

What will happen if I change the distribution (block size, halo layer) of a existing Matrix ? Will the DASH support re-distribution the data following new parameters?

I'm not sure that is possible with the current interface. Once a matrix is allocated there is no way to change its layout (afaics).

devreal avatar Mar 03 '20 09:03 devreal

Hi @devreal Got it and thanks for the clarification.

Bests, Bin

Goon83 avatar Mar 04 '20 05:03 Goon83

Could you elaborate what you have in mind for it? Maybe we will get a chance at some point to implement such a feature if there is a good use case.

devreal avatar Mar 04 '20 07:03 devreal

Hi @devreal It is like the "repart" operator in SciDB (doc). . The "repart" changes the chunk size (called tile size in DASH) of an existing array. The "repart" in SciDB mostly involves the disk operation (might be just metadata but not sure so far). A "repart" for DASH array definitely needs to consider lots of communication among processes (units in DASH) to redistribute data.

We might also explore this idea in near future. To avoid repeating work, ping me [email protected] if you are interested in working together. We can have a chat then.

Bests, Bin

Goon83 avatar Mar 06 '20 18:03 Goon83

@dash-project/developers First, I want to announce my "comeback" as an active contributor to DASH after my hiatus of over a year. Regarding this feature request, I have prepared concepts for dynamic adaptations such as load-balancing dash::Matrix (aka dash::NArray) which I originally designed. Also, I'm currently supervising a master's thesis on sparse matrix concepts for DASH. I will put the respective current states of development in feature branches as soon as possible.

In brief, the NArray / Matrix concepts will not be affected but extended by additional free-function-style concepts. So if you (@Goon83) can use dash::Matrix in its current state and postpone the rebalancing / repartitioning parts in your work, your existing code will be unaffected.

fuchsto avatar Apr 25 '20 10:04 fuchsto

Hi @fuchsto Sorry for not monitoring this thread closely.

Welcome back to this wonderful project.

Yes, the rebalancing feature is very interesting to us.

Let me know how can I help. Bests, Bin

Goon83 avatar May 28 '20 17:05 Goon83