Support MPI for CUDA and HIP when not GPU-aware

[sslattery]

We should allow for versions of MPI to be used that aren't GPU-aware. My proposal to do this is to add an extra optional template parameter to the communication classes which describes the communication memory space. We then simply mirror the user data into the communication memory space prior to calling MPI.

Some sensible defaults will be assuming CUDA-aware MPI and using the host space for HIP. A user can always change this in their implementation depending on their dependencies. We could then add a CMake option which would change these defaults for the user so they could easily configure for the given system they are using.

Once a HIP-aware MPI becomes available we can use this as the HIP default.

[rfbird]

bump on this, I just ran into it when making sure everything passed but I wasn't using an GPU aware MPI

[streeve]

Very stale and I don't think necessary - a manual copy to the host will work if really necessary (as in the comm benchmarks)