Initial partition based particle density profile

[WeiqunZhang]

trafficstars

This adds the option to do initial partition based on particle density profile so that none of the Boxes has too many particles.

Two new runtime parameters warpx.init_partition_species and warpx.init_partition_efficiency are introduced. If warpx.init_partition_species is set (e.g., to electron), we will query electron.profile, electron.parse_density_function and electron.density_min. If all these parameters are present, we will try to divide the original BoxArray until no Box has more particles than the threshold defined as the total number of particles divided by the number of processes and then multiplied by warpx.init_partition_efficiency.

[n01r]

Thanks a lot, @WeiqunZhang! I'll give this a try! :tada:

[n01r]

Hi @WeiqunZhang,

The result looks really good! At step 0, we already have 56.7% load balance efficiency.

At step 100, after the first load-balancing iteration, we went up to 88.4%.

There are these artifacts still in there, I think (if they don't come from my plotting), where we get extremely long boxes vertically. Do you see where they might come from?

[WeiqunZhang]

I don't see any reason that we would get extremely long boxes.

[n01r]

@WeiqunZhang, is it possible to add a "fake" density profile of a species that we won't actually initialize? Can we use this to anticipate the load later in the simulation?

[ax3l]

@WeiqunZhang Thank you for this PR!

As a next step to get KISMET and INCITE sims like those to run, I think what we need is a functionality to dynamically split boxes during simulation runtime, if too many particles move into it.

Merging boxes again is another challenge for later, we did something along these lines in I/O a while back where we can lend an algorithm from: https://doi.org/10.1109/TPDS.2021.3100784

[ax3l]

@n01r Can you pls share on Slack or email the inputs files for the scenario benchmarked here?

I think this PR makes perfect sense and we now want to work on a follow-up where we dynamically split the boxes behind the target once particles start moving into them :)

[n01r]

@n01r Can you pls share on Slack or email the inputs files for the scenario benchmarked here?

A copy of the corresponding directory with the input deck is here (NERSC): /global/cfs/cdirs/m4546/mgarten/projects/kismet/2025/043_256_node_test_Weiqun_partitioning/

Alternatively, here's a GDrive link that will only work for LBL-internal folks https://drive.google.com/drive/folders/1KL6uOKMUCsljRcCRw620VCZV5a8ETniW?usp=sharing

[ax3l]

I think I found a bug in a smaller example (#6107) #6158 , which might help us understand these large boxes we saw above, too.