EKAT
EKAT copied to clipboard
Remove assumption of CudaUVM in order to create Host team policies
PR #218 added the ability to dispatch a kernel on host when CudaUVM is used, by creating a team policy on host. However, CudaUVM is still a device memory space, with behind-the-scenes sync between host and device. In particular, this means that a View with memory space HostSpace
cannot be assigned to a view with memory space CudaUVM
, and viceversa. This can have bad consequences for codes that use host mirrors, and are templated on the memory space. For instance, the following won't work:
template<typename MemSpace>
struct Types {
using view = Kokkos::View<double*, MemSpace, Kokkos::LayoutRight>;
};
typename Types<Kokkos::CudaUVM>::view my_view_d ("",10);
typename Types<HostSpace>::view my_view_h = Kokkos::create_host_mirror(my_view_d); // ERROR!
since the host mirror still has CudaUVM as mem space (b/c uvm is accessible from host), but CudaUVM is not assignable to HostSpace. On the other hand, CudaHostPinnedSpace is both assignable and accessible from device and host. The snippet above would work with CudaHostPinnedSpace
in place of CudaUVM
.
However, notice that nothing in the team policy requires to know what memory space the views that will be accessed will be on. In fact, the concept of "view" is orthogonal to that of a team policy. All that the team policy needs to know is the execution space where the kernel will be run on. We should therefore remove the requirement that CudaUVM is used, and allow the user to create a policy on host regardless. It is up to the user to ensure that the views used in the kernel are accessible on host.
The drawback would be that Kokkos does not offer a configuration option to make CudaHostPinnedSpace
the default mem space for the Cuda
execution space (while it does so for CudaUVM
). Therefore, any usage of ExecSpace::memory_space
would yield something different from the host pinned space.