WarpX
WarpX copied to clipboard
ECP-WarpX
Problems with injection from external file
Hi all! I am trying to initialize particles from an external file to get a custom distribution with aggregates of particles. When I increase the number of particles in the openPMD file above a certain level, the simulation crashes with the error out of GPU memory. For example, it works with an openPMD file of 7,6GB, but it crashes with one of 8.8GB. However, a simulation with the same resources but using random injection without an external file can easily load the same number of particles. Increasing resources does not help, so I suppose there are some limits due to the loading from the file. Am I right? Is this problem solvable? I am working on Marconi100 with the latest version of WarpX. Input file: input0.txt Job: job.txt Output files: Backtrace.0.txt job.err.txt output.txt
This problem still exists with the latest version of Warpx. I investigated if a PICMI input could help in initializing such arbitrary densities, but I didn't find a feasible solution. Do you have any suggestions?
Thank you for the detailed report.
The issue appears because the external file read logic for particles is not yet MPI-parallelized: https://github.com/ECP-WarpX/WarpX/blob/23.06/Source/Particles/PhysicalParticleContainer.cpp#L576-L593 I think the reason for that is that people used this function so far only to load particle beams (usually a few 10k particles) and not full targets/background plasma distributions.
The way to update this is:
- loop over smaller, unique chunks of particles (e.g., 100M max at a time) per MPI rank
We do similar logic in regular initialization of plasma profiles.
When we add particles, we automatically redistribute them on the correct rank as long as the position is valid in the simulation box (otherwise they get removed).
Do you like to contribute this logic to the linked location or do you like to wait until someone has time to give it a try on our end? :)
Hi @ax3l, Thank you for answering my issue, I want to try to contribute following your indications. However, if I see that I have many difficulties in doing it, I will wait for someone more expert to solve it. I will update you on my attempt.
Hi @MartaGalbi, I was just wondering have you had any success with distributing the particle initialisation? Thanks, Daniel
