warpx.roundrobin_sfc: A runtime parameter to control distribution mapping

[WeiqunZhang]

The default is false. If it's true, AMReX's RRSFC strategy will be used to override the default SFC strategy used by amrex::AmrCore. The motivation for this is that this might mitigate the load imbalance issue during initialization by avoiding putting neighboring boxes on the same process.

[dpgrote]

Looks good. Would his affect the runtime, slowing the MPI communication since neighboring boxes will be far away from each other?

[ax3l]

@WeiqunZhang sorry for the delay, can you please rebase?

[n01r]

Thank you, @WeiqunZhang! I will test/use this down the line. :)

[ax3l]

Thank you so much!

After checking back with our collaborators, we currently do not have a reproducer at hand, but I'll include it in the coming release so we can try it when we should find one again.

@n01r might have one we can try this on, too.

[ax3l]

@dpgrote

Looks good. Would his affect the runtime, slowing the MPI communication since neighboring boxes will be far away from each other?

The idea is to overcome out-of-memory in the init routines (e.g., for LPI sims with a lot of particles in close-by ranks) and then apply load balancing based on a user-defined cost (and distribution) model, as usual.