Tribol icon indicating copy to clipboard operation
Tribol copied to clipboard

Initial support for GPUs in Redecomp and the Tribol MFEM interface

Open ebchin opened this issue 1 year ago • 1 comments

This PR adds initial support for GPU data in the Redecomp library using MFEM's built in GPU support. This PR adds initial plumbing and tests on GPU to ensure GPU data transfers in Redecomp are functioning correctly. Follow-on PRs will add additional GPU support in Tribol and Redecomp.

Transfers from parallel data (grid functions and quadrature functions) on device to redecomp data are done as follows:

  1. Device parallel data is copied to host using mfem::Vector::HostRead().
  2. MPI communication moves host data from parallel rank to its redecomp rank.
  3. Host redecomp data is written to a host pointer obtained using mfem::Vector::HostWrite().

The final result is redecomp data on the host, which can be copied to the GPU as needed in MFEM. Transfers from redecomp data to parallel data are done in reverse order.

The support for GPU data is extended to the Tribol MFEM interface, which leverages the Redecomp library to prepare data for processing by Tribol. Since Tribol does not support GPUs, redecomp data is left on host where Tribol can compute contact contributions.

GPU support has been added to the following examples (and corresponding tests):

  1. domain_redecomp.cpp: use the Redecomp library to transfer parallel data on device to redecomp data on device and back
  2. mfem_common_plane.cpp: multirank common plane contact using MFEM (with GPU) and Tribol
  3. mfem_mortar_lm_patch.cpp: multirank contact patch test using MFEM (with GPU) and Tribol mortar

ebchin avatar Jan 30 '24 19:01 ebchin