EKAT icon indicating copy to clipboard operation
EKAT copied to clipboard

Remove assumption of CudaUVM in order to create Host team policies

Open bartgol opened this issue 2 years ago • 0 comments

PR #218 added the ability to dispatch a kernel on host when CudaUVM is used, by creating a team policy on host. However, CudaUVM is still a device memory space, with behind-the-scenes sync between host and device. In particular, this means that a View with memory space HostSpace cannot be assigned to a view with memory space CudaUVM, and viceversa. This can have bad consequences for codes that use host mirrors, and are templated on the memory space. For instance, the following won't work:

template<typename MemSpace>
struct Types {
  using view = Kokkos::View<double*, MemSpace, Kokkos::LayoutRight>;
};

typename Types<Kokkos::CudaUVM>::view  my_view_d ("",10);
typename Types<HostSpace>::view my_view_h = Kokkos::create_host_mirror(my_view_d); // ERROR!

since the host mirror still has CudaUVM as mem space (b/c uvm is accessible from host), but CudaUVM is not assignable to HostSpace. On the other hand, CudaHostPinnedSpace is both assignable and accessible from device and host. The snippet above would work with CudaHostPinnedSpace in place of CudaUVM.

However, notice that nothing in the team policy requires to know what memory space the views that will be accessed will be on. In fact, the concept of "view" is orthogonal to that of a team policy. All that the team policy needs to know is the execution space where the kernel will be run on. We should therefore remove the requirement that CudaUVM is used, and allow the user to create a policy on host regardless. It is up to the user to ensure that the views used in the kernel are accessible on host.

The drawback would be that Kokkos does not offer a configuration option to make CudaHostPinnedSpace the default mem space for the Cuda execution space (while it does so for CudaUVM). Therefore, any usage of ExecSpace::memory_space would yield something different from the host pinned space.

bartgol avatar Jun 24 '22 21:06 bartgol