nalu-wind icon indicating copy to clipboard operation
nalu-wind copied to clipboard

Published 20 hours ago • verified publisher icon Exawind

Nalu-Wind Kernels – GPU transition

Open sayerhs opened this issue 5 years ago • 7 comments

Tracking the progress of conversion of Nalu-Wind kernels for execution on GPUs. The basic process of converting a non-HO Kernel to GPU can be seen in commits 1e9382f66f4823 (volume master element) and 5ee617e3ad555 (surface master element). The steps are:

  • Switch base class from Kernel to NGPKernel<T<AlgTraits>> where T = MomentumMassElemKernel for example.
  • Replace private/deleted default constructor with a default public constructor that has the KOKKOS_FUNCTION decorator.
  • Add KOKKOS_FUNCTION decorator to destructor, and use default where possible.
  • Remove all non-POD data types from the Kernel, i.e., remove Kokkos::View and any references (e.g., CoriosSrc stores a reference to SolutionOptions, this needs to be removed).
  • Remove any pointers to stk::mesh::FieldBase and replace them with field ordinals.
  • Switch the statement getting meSCS and meSCV pointer to use the templated get_surface_master_element and get_volume_master_element version; store a reference to this pointer in the instance variables for access in Kernel::execute method
  • Replace shape function logic by registering SCS_SHAPE_FCN to ElemDataRequests and accessing it via ScratchViews in Kernel::execute implementation. Access adjacentNodes and nodeMap variables via the meSCS_ and meSCV_ instances in the execute method.
  • Add DeviceShmem and DeviceTeamHandleType template parameters to arguments to the Kernel::execute method.
  • Convert the unit test and add an NGP_ prefix for easy filtering.
# Run only NGP-ready unit tests on GPU
$ ./unittest --gtest_filter=*.NGP*

Note that #186 may need to be resolved before some of the Kernels can be fully converted.

Non high-order kernels

  • [x] AssembleElemSolverAlgorithm (5016c5e208, c553311966)
  • [x] AssembleFaceElemSolverAlgorithm (#219)

Interior algorithms; ready for conversion

  • [x] kernel/ContinuityAdvElemKernel (#194)
  • [x] kernel/ContinuityMassElemKernel (#188)
  • [x] kernel/MomentumAdvDiffElemKernel (#195)
  • [x] kernel/MomentumMassElemKernel (#195)
  • [x] kernel/ScalarMassElemKernel (#203)
  • [x] kernel/ScalarAdvDiffElemKernel (#200)
  • [x] kernel/TurbKineticEnergySSTDESSrcElemKernel (#212)
  • [x] kernel/TurbKineticEnergySSTSrcElemKernel (#209)
  • [x] kernel/SpecificDissipationRateSSTSrcElemKernel (#211)
  • [x] kernel/WallDistElemKernel (#204)

Requires additional infrastructure (addressed in #220)

  • [x] kernel/MomentumUpwAdvDiffElemKernel
  • [x] kernel/ScalarUpwAdvDiffElemKernel (#220)

Boundary conditions (ready for conversion after #219)

  • [x] kernel/ContinuityInflowElemKernel (#417)
  • [ ] kernel/ContinuityOpenElemKernel
  • [ ] kernel/MomentumOpenAdvDiffElemKernel
  • [ ] kernel/MomentumSymmetryElemKernel
  • [ ] kernel/MomentumWallFunctionElemKernel
  • [ ] kernel/ScalarOpenAdvElemKernel

Interior algorithms (2); ready for conversion

  • [x] kernel/MomentumActuatorSrcElemKernel (#216)
  • [x] kernel/MomentumBuoyancyBoussinesqSrcElemKernel (#217)
  • [x] kernel/MomentumBuoyancySrcElemKernel (#218)
  • [x] kernel/MomentumCoriolisSrcElemKernel (reference to SolutionOptions needs to be removed)
  • [x] kernel/TurbKineticEnergyKsgsSrcElemKernel (#213)
  • [x] kernel/TurbKineticEnergyKsgsDesignOrderSrcElemKernel (#214)
  • [x] kernel/MomentumHybridTurbElemKernel (#215)
  • [ ] nso/MomentumNSOElemKernel
  • [ ] nso/MomentumNSOKeElemKernel
  • [ ] nso/MomentumNSOSijElemKernel
  • [ ] nso/ScalarNSOElemKernel
  • [ ] nso/ScalarNSOKeElemKernel
  • [ ] kernel/ScalarDiffElemKernel
  • [ ] kernel/ScalarDiffFemKernel
  • [ ] kernel/ScalarFluxPenaltyElemKernel
  • [ ] user_functions/SteadyThermal3dContactSrcElemKernel
  • [ ] pmr/RadTransAbsorptionBlackBodyElemKernel
  • [ ] pmr/RadTransAdvectionSUCVElemKernel
  • [ ] pmr/RadTransIsotropicScatteringElemKernel
  • [ ] pmr/RadTransWallElemKernel

HO Kernels

Needs HO master elements

  • [ ] AssembleElemSolverAlgorithmHO
  • [ ] kernel/ContinuityMassHOElemKernel
  • [ ] kernel/MomentumAdvDiffHOElemKernel
  • [ ] kernel/MomentumBuoyancySrcHOElemKernel
  • [ ] kernel/MomentumMassHOElemKernel
  • [ ] kernel/PressurePoissonHOElemKernel
  • [ ] kernel/ProjectedNodalGradientHOElemKernel
  • [ ] kernel/ScalarAdvDiffHOElemKernel
  • [ ] kernel/ScalarDiffHOElemKernel
  • [ ] kernel/ScalarMassHOElemKernel
  • [ ] user_functions/SteadyThermalContactSrcHOElemKernel
  • [ ] user_functions/TGMMSHOElemKernel
  • [ ] user_functions/VariableDensityContinuityMMSHOElemKernel
  • [ ] user_functions/VariableDensityEnthalpyMMSHOElemKernel
  • [ ] user_functions/VariableDensityMomentumMMSHOElemKernel

sayerhs avatar Apr 21 '19 16:04 sayerhs

I'm currently working on kernel/ScalarMassElemKernel. (It's done and passing regression tests, but I don't have it unit-tested yet.)

alanw0 avatar Apr 21 '19 18:04 alanw0

@alanw0 Thanks, I'll use the [x] box to indicate someone is working on the kernel and then add the commit/pull-request ID to mark the kernel as DONE.

sayerhs avatar Apr 21 '19 19:04 sayerhs

AssembleFaceElemSolverAlgorithm in progress by @alanw0

sayerhs avatar Apr 22 '19 18:04 sayerhs

I am working on kernel/MomentumUpwAdvDiffElemKernel

ashesh2512 avatar Apr 22 '19 18:04 ashesh2512

I think I got kernel/TurbKineticEnergySSTSrcElemKernel with #209 . I'm going to try the rest of the SST kernels as well.

jrood-nrel avatar Apr 30 '19 16:04 jrood-nrel

A note for when we return to element-based GPU conversion: Regarding AssembleFaceElemSolverAlgorithm. The run_algorithm method is converted and unit-tested, but the execute method needs to be converted next. Need to convert the unit-tests in UnitTestFaceElemBasic.C which call faceElemAlg.execute().

alanw0 avatar May 09 '19 22:05 alanw0

#829 removed these kernels

sayerhs avatar Mar 27 '21 17:03 sayerhs