eckit
eckit copied to clipboard
eckit::linalg::sparse::LinearAlgebraTorch backend
This PR adds a sparse lienar algebra backend to allow for GPU-based matrix multiplications (and other operations), which translates into a significant performance increase for interpolations, in the right conditions (right environment, advanced use of mir.)
It makes use of a deployed version of PyTorch (findable by CMake), specifically its lower level component "Torch", which is part of the same package (this is how it is released to the public.) I've exposed all possible hardware configuration options, contemporary. But obviously, the better development is to improve the whole workflow to avoid copies to/from the CPU/GPU, so this develolpment is purelly a stepping stone -- it has already allowed me to run both mars-client (C) and pgen on GPUs, and of course mir. It would be great to follow this up with a publication. Possibly, this could be configurable in earthkit-regrid for maximum marketing :-)
I've held back this development for several months, but I couldn't find a definite response on when to post it -- here it is.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 64.09%. Comparing base (
f1591f4) to head (b548c96).
Additional details and impacted files
@@ Coverage Diff @@
## develop #165 +/- ##
===========================================
- Coverage 64.09% 64.09% -0.01%
===========================================
Files 1083 1083
Lines 55759 55759
Branches 4120 4120
===========================================
- Hits 35738 35736 -2
- Misses 20021 20023 +2
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
This looks neat! I'd like to test this before merging, preferably on our HPC. Could you get a working Torch that I could link into e.g. with a setup using the nvidia compiler?
As a side, it would be nice to clean up and remove unused backends (armadillo, viennacl, ... )