dash icon indicating copy to clipboard operation
dash copied to clipboard

Set default window type to non-shared static

Open devreal opened this issue 6 years ago • 3 comments

This PR changes the build scripts and CMakeList to configure DART to use regular windows instead of dynamic + shared memory windows in dart_team_memalloc_aligned.

Background: The use of dynamic windows causes all sorts of troubles:

  1. It prevents the use of RDMA capabilities on any decent hardware.
  2. Open MPI chokes if MPI_THREAD_MULTIPLE is enabled and no RDMA-capable network is found (thanks to @bertwesarg for bringing this back up) because the shared-memory component does not support dynamic windows (how could it?) and the pt2pt component lacks support for MPI_THREAD_MULTIPLE (not sure this will ever be fixed). Hence, DASH won't even work on a single node.

The MPI implementation will likely short-cut through shared memory for on-node communication so the benefit of doing it ourselves really seems marginal. On the other hand, changing the default avoids some bad surprises.

devreal avatar Mar 25 '19 20:03 devreal

Codecov Report

Merging #639 into development will decrease coverage by 0.02%. The diff coverage is n/a.

@@               Coverage Diff               @@
##           development     #639      +/-   ##
===============================================
- Coverage        84.94%   84.92%   -0.03%     
===============================================
  Files              335      335              
  Lines            24821    24821              
  Branches         11236    11709     +473     
===============================================
- Hits             21085    21080       -5     
  Misses            3733     3733              
- Partials             3        8       +5
Impacted Files Coverage Δ
dash/include/dash/Team.h 87.36% <0%> (-5.27%) :arrow_down:

codecov[bot] avatar Mar 25 '19 20:03 codecov[bot]

I would not consider the build*.sh scripts a good place to document CMake variables. why not inside CMakeLists.txt into the options()?

Btw, does other combinations instead of off&off or on&on make sense too?

bertwesarg avatar Mar 26 '19 13:03 bertwesarg

I would not consider the build*.sh scripts a good place to document CMake variables. why not inside CMakeLists.txt into the options()?

We can move it to the options in the CMake file. So far these descriptions are kept quite short though and the description of these variables would have to be split into two.

Btw, does other combinations instead of off&off or on&on make sense too?

There is one case where dynamic windows without shared windows make sense. That is related to #615: dynamic windows do not register the memory and thus don't fault the pages, leading to better NUMA behavior. There are also some MPI implementations that don't really do well with shared memory window IIRC. So as long as we don't have the NUMA issue under control I'd leave the two options separate but we should consider merging them at some point.

devreal avatar Mar 27 '19 13:03 devreal