WarpX icon indicating copy to clipboard operation
WarpX copied to clipboard

[WIP]Add plasma particles relative to problo instead of local block

Open dpgrote opened this issue 1 year ago • 2 comments

This is a small simplification when adding plasma particles. The code was calculating the particle positions using indices that are relative to the local box. This required the extra shifted vector. This PR instead uses indices relative to the global domain (adding that shifted vector to the overlap_box), allowing a simplification in the code.

An advantage of this method is that since the positions are calculated relative to the global domain, they will be consistent independent of the block decomposition, and not subject to differences in round-off. This can be helpful when comparing simulations with differing number of processors for example.

Note that some benchmarks needed updating because of differences in round off in the particle initial positions.

dpgrote avatar May 12 '23 16:05 dpgrote

Thanks Dave!

I was thinking earlier today about the opposite direction and added some notes to our developer meeting. Let's chat on Tue which strategy to follow towards SP particle positions.

My notes so far for Tue are:

  • Particle positions & single precision [Axel]
    • Problem:
      • We cannot yet run in SP for many sims, due to accumulating errors in field gather and current/charge deposition at large offsets, non-zero-centered sim boxes, moving windows, etc.
    • What would it take to make particle positions relative to the simulation lower box?
      • Idea based on discussion with @AlexanderSinn and used in HiPACE++.
      • Alternatively, in PIConGPU, we calculated relative to the supercell (could be our AMReX block) and cell index.
    • Would this be sufficient for still having a pretty precise in-cell position for particles in the highest sim-box cell at every step?

ax3l avatar May 18 '23 18:05 ax3l

@ax3l In general terms, this PR is probably not needed. I only needed this consistency since I'm working on tracking down a problem in the implicit implementation that only occurs with many processes. I want to compare to machine precision the results on differing number of processors. I submitted the PR in case others might want this consistency. It turns out though that the machine precision agreement is lost on the first step since the current density ends up different since the order of the sums changes for cells near the boundaries.

If we can use the method of PIConGPU where the particle positions are relative to the cell index, this would also remove the issue this PR addresses and the particles would be identical independent of the number of processors.

dpgrote avatar May 18 '23 18:05 dpgrote