Nalu-Wind Edge Algorithms – GPU Transition

[sayerhs]

Track progress of the conversion of Edge algorithms in Nalu-Wind to be capable of running on GPUs.

Edge Interior algorithms

Status: Ready for conversion

Basic implementation of the NGP-version of interior edge algorithm was demonstrated in #232, see ContinuityEdgeSolverAlg

~Outstanding issues~: Momentum and Scalar AdvDiff algorithm will require non-SIMD versions of ngp_create_peclet_function – should be a templated function like the non-NGP version. Fixed in d2157ff60d67, use

eqSystem->ngp_create_peclet_function<double>(velocity->name())

• [x] AssembleContinuityEdgeSolverAlgorithm => edge_kernels/ContinuityEdgeSolverAlg (#232)
• [x] AssembleMomentumEdgeSolverAlgorithm (#267)
• [x] AssembleScalarEdgeSolverAlgorithm (#271)
• [x] AssembleWallDistEdgeSolverAlgorithm (#241)
• [x] AssembleNodalGradEdgeAlgorithm
• [x] AssembleNodalGradUEdgeAlgorithm
• [ ] AssembleScalarEdgeDiffSolverAlgorithm
• [ ] AssembleScalarEigenEdgeSolverAlgorithm

Edge Boundary algorithms

Status: Ready for conversion

These algorithms will require access to meSCS for the element connected to the face, so will require a hybrid approach where we do use scratch views to assemble the element data and be templated on element traits.

• [x] AssembleMomentumEdgeSymmetrySolverAlgorithm => edge_kernels/MomentumSymmetryEdgeKernel (#272)
• [x] AssembleMomentumEdgeABLWallFunctionSolverAlgorithm (#274)
• [x] AssembleTemperatureNormalGradientBCSolverAlgorithm (#278)
• [x] AssembleScalarEdgeOpenSolverAlgorithm (#306)
• [x] AssembleContinuityEdgeOpenSolverAlgorithm (#417)
• [x] AssembleMomentumEdgeOpenSolverAlgorithm (#421)
• [ ] AssembleMomentumEdgeABLTopBC
• [ ] AssembleMomentumEdgeWallFunctionSolverAlgoirthm

Node source supplemental algorithms

Status: Ready for conversion

The infrastructure necessary for converting NodeSuppAlg (or NodeSrcSuppAlg) has been implemented in #243, and follows the general steps for element kernels (#202). However, these will require the creation of the new NodeKernel classes. See MomentumBodyForce as an example (NGP version and NodeSrcSuppAlg version).

In the NGP-version, we end up doing some extra work compared to the NodeSuppAlg version, specifically creating a copy of the ngp::Field instances used in execute method while storing the FieldOrdinal in the constructor. This is necessary for cases with mesh motion where the ngp::Field instance has to be created from scratch because of changes in ngp::Mesh instance. Handling this in setup() method, will get the correct ngp::Field instance for use in execute() method. For the time derivative algorithms, you will also have to update dt, gamma{1,2,3} etc. within the setup() method.

Time derivative terms

• [x] ContinuityMassBDF2NodeSuppAlg(#259)
• [x] MomentumMassBDF2NodeSuppAlg (#248)
• [x] ScalarMassBDF2NodeSuppAlg (#247)
• [x] ContinuityMassBackwardEulerNodeSuppAlg(consolidated into #259)
• [x] MomentumMassBackwardEulerNodeSuppAlg(consolidated into #248)
• [x] ScalarMassBackwardEulerNodeSuppAlg(consolidated into #247)

Geometric conservation law terms

• [x] ContinuityGclNodeSuppAlg (#282)
• [x] MomentumGclSrcNodeSuppAlg (#283)
• [x] ScalarGclNodeSuppAlg (#285)

Source terms

• [x] EnthalpyABLSrcNodeSuppAlg (#266)
• [x] MomentumABLForceSrcNodeSuppAlg (#266)
• [x] MomentumCoriolisSrcNodeSuppAlg(#253)
• [x] TurbKineticEnergyKsgsNodeSourceSuppAlg (#279)
• [x] MomentumActuatorSrcNodeSuppAlg=>(#254 )
• [x] MomentumBodyForceSrcNodeSuppAlg => (#243) node_kernels/MomentumBodyForceNodeKernel
• [x] MomentumBoussinesqSrcNodeSuppAlg (#246)
• [x] SpecificDissipationRateSSTNodeSourceSuppAlg (#279)
• [x] TurbKineticEnergyRodiNodeSourceSuppAlg (#269)
• [x] TurbKineticEnergySSTDESNodeSourceSuppAlg (#279)
• [x] TurbKineticEnergySSTNodeSourceSuppAlg (#279)
• [x] WallDistSrcNodeSuppAlg(#253)

---

• [ ] HeatCondMassBackwardEulerNodeSuppAlg
• [ ] HeatCondMassBDF2NodeSuppAlg
• [ ] ContinuityLowSpeedCompressibleNodeSuppAlg
• [ ] EnthalpyLowSpeedCompressibleNodeSuppAlg
• [ ] EnthalpyPressureWorkNodeSuppAlg
• [ ] EnthalpyViscousWorkNodeSuppAlg
• [ ] MomentumBoussinesqRASrcNodeSuppAlg
• [ ] MomentumBuoyancySrcNodeSuppAlg
• [ ] mesh_motion/MeshDisplacementMassBackwardEulerNodeSuppAlg
• [ ] user_functions/BoussinesqNonIsoEnthalpySrcNodeSuppAlg
• [ ] user_functions/BoussinesqNonIsoMomentumSrcNodeSuppAlg
• [ ] user_functions/SteadyTaylorVortexMomentumSrcNodeSuppAlg
• [ ] user_functions/SteadyThermalContact3DSrcNodeSuppAlg
• [ ] user_functions/SteadyThermalContactSrcNodeSuppAlg
• [ ] user_functions/VariableDensityContinuitySrcNodeSuppAlg
• [ ] user_functions/VariableDensityMixFracSrcNodeSuppAlg
• [ ] user_functions/VariableDensityMomentumSrcNodeSuppAlg
• [ ] user_functions/VariableDensityNonIsoContinuitySrcNodeSuppAlg
• [ ] user_functions/VariableDensityNonIsoEnthalpySrcNodeSuppAlg
• [ ] user_functions/VariableDensityNonIsoMomentumSrcNodeSuppAlg

[sayerhs]

@marchdf @jamelvin Please add details on the TAMS edge implementation here, so that we can track that also.

[marchdf]

The TAMS branch currently lives here: https://github.com/marchdf/nalu-wind/tree/TAMS-Hybrid

The following are element kernels that we implemented that we are going to need to convert to edge kernels:

• [x] src/kernel/MomentumTAMSKEpsDiffElemKernel.C
• [x] src/kernel/MomentumTAMSSSTDiffElemKernel.C
• [x] src/kernel/ScalarTAMSAdvDiffElemKernel.C (done by removing and hooking into existing infrastructure)
• [x] src/kernel/ScalarTAMSUpwAdvDiffElemKernel.C (done by removing and hooking into existing infrastructure)

The KEps versions will probably not make it back into nalu-wind master (they are for our debugging purposes).

[sayerhs]

@jrood-nrel @alanw0 @overfelt The ABLSrc terms need a little more surgery and can't be converted by just implementing a NodeKernel. I'll have some modifications to the classes in wind_energy before we can implement that.

[sayerhs]

Status update [July 4, 2019]

Almost all nodal and edge assembly algorithms necessary to perform an ABL precursor simulation have been converted to NGP (once #278 and #279 gets merged). The first round of integration of these algorithms with EquationSystems is pending in #277. Second round of integration after pending pull requests are merged.