Separate RAJA execution policies specification from linear algebra classes implementations

[pelesh]

Linear algebra classes that use preprocessor directives to specify RAJA execution policies. This may become cumbersome as we add more execution policies (CUDA, HIP, etc.). Furthermore, each linear algebra object implementation duplicates part of policy specifications adding some maintenance overhead. Consider specifying RAJA policies within a container class in a separate source file.

CC @ashermancinelli @CameronRutherford @nychiang @cnpetra

Potential additional benefits: In addition to making linear algebra code more compact, this could simplify creation of linear algebra class templates parametrized by execution type. See #241 for more details. Currently, same execution policies are set at configure time for all linear algebra objects in HiOp.

[pelesh]

Possible solution could be to create policy specification containers like this:

// hiopExecPolicies.hpp

#include <hiop_defs.hpp>

#ifdef HIOP_USE_RAJA

#ifdef HIOP_USE_GPU
struct hiopCudaPolicies
{
  hiopExecutionPolicies() = default;
  using hiop_raja_exec   = RAJA::cuda_exec<HIOP_CUDA_BLOCK_SIZE>;
  using hiop_raja_reduce = RAJA::cuda_reduce;
  using hiop_raja_atomic = RAJA::cuda_atomic;
  namespace policy_namespace = RAJA;
};
#endif // HIOP_USE_GPU

struct hiopOmpPolicies
{
  hiopExecutionPolicies() = default;
  using hiop_raja_exec   = RAJA::omp_parallel_for_exec;
  using hiop_raja_reduce = RAJA::omp_reduce;
  using hiop_raja_atomic = RAJA::omp_atomic;
  namespace policy_namespace = RAJA;
};

#endif // HIOP_USE_RAJA

Then in, for example, hiopVectorRajaPar class, one can simplify the preprocessor directives like this:

// hiopVectorRajaPar.cpp

#include <hiopExecPolicies.hpp>

// some code

#ifdef HIOP_USE_GPU  
#include "cuda.h"
#define RAJA_LAMBDA [=] __device__
using hiop_policy_type = hiopCudaPolicies;
#else  
#define RAJA_LAMBDA [=]
using hiop_policy_type = hiopOmpPolicies;
#endif

using hiop_raja_exec   = hiop_policy_type::hiop_raja_exec;
using hiop_raja_reduce = hiop_policy_type::hiop_raja_reduce;
using hiop_raja_atomic = hiop_policy_type::hiop_raja_atomic;
namespace policy_namespace = hiop_policy_type::policy_namespace;

// more code ...

[ashermancinelli]

I think these are great ideas, but what's the point of policy_namespace? Is this foreshadowing some hiopForAll function that doesn't use a raja policy?

[pelesh]

I think these are great ideas, but what's the point of policy_namespace? Is this foreshadowing some hiopForAll function that doesn't use a raja policy?

Yes, indeed. This could allow for replacing RAJA function forall and objects Range, ReduceSum, etc. with those from HiOp namespace. With that, you could create RAJA and non-RAJA linear algebra objects from the same code by just passing appropriate template parameter. At least, that is the idea.

[cnpetra]

closed by https://github.com/LLNL/hiop/pull/543 with a couple of small, follow-up action items tracked by https://github.com/LLNL/hiop/issues/574