amrex icon indicating copy to clipboard operation
amrex copied to clipboard

SumBoundary is much less efficient than FillBoundary

Open AlexanderSinn opened this issue 1 year ago • 1 comments

If I understand the code correctly, FillBoundary only iterates over cells close to the boundary, while SumBoundary copies the entire Field into a temporary array and then copies it back with updated boundaries, wich is not nessisary for the vast majority of cells in the center of the box. With one rank and one box SumBoundary is much slower than FillBoundary. In HiPACE++ I could make a custom single-box local SumBoundary version, but maybe there is a better option from AMReX?

For context: HiPACE++ currently uses FillBoundary for charge and currents, which is wrong, but changing it to SumBoundary increases the total runtime by 5%.

AlexanderSinn avatar Jan 31 '24 16:01 AlexanderSinn

The current implementation of SumBoundary tries to reuse what we have. And the thinking was for large jobs, the MPI communication cost will dominate the local copy cost. We could reimplement the function for efficiency .

WeiqunZhang avatar Jan 31 '24 16:01 WeiqunZhang