amrex
amrex copied to clipboard
WriteMultiLevelPlotfile copies data from Pinned to Device memory
In our AMReX-based code Hipace, we allocate the main 3D MultiFab in pinned memory, but when we call WriteMultiLevelPlotfile on this MultiFab it seems to be copied to the device memory. This is limiting because this MultiFab is much larger than the device memory for our production simulations, so it makes the simulation crash. See https://github.com/Hi-PACE/hipace/issues/145 for a full description.
Update: this only happens if the MultiFab has guard cells. Otherwise, things work as expected.