CEMeNT-PSAAP
GPU regression testing and reproducibility
Currently, GPU results do not exactly reproduce CPU results for regression tests. This seems to be because of an issue with numba's memory re-allocation (#224, also here).
For now, we plan to create GPU-specific answer files to:
- [ ] ensure GPU reproducibility within a single architecture when code changes are made,
- [ ] ensure GPU reproducibility across architectures.
If numba were to change how it handles the memory issue, that could break our reproducibility; that would also be the case for CPU, and that hasn't happened yet. For future numba version releases, we should check back with this issue before supporting the new versions.
I want to make sure: (1) the issue (#224) also applies to the current CPU mode and (2) that GPU mode does not reproduce the CPU mode results may be due to other issues, right @braxtoncuneo?
GPU is found to reproduce SOME of the CPU results. The following may be a useful reference to help point out what breaks the CPU-GPU reproducibility:
cc: @braxtoncuneo, @jpmorgan98
I attempted to create the GPU regression test keys here: https://github.com/ilhamv/MCDC/tree/gpu_regression_test
However, I found that the current GPU implementation is not reproducing its own solution for two problems:
-
inf_shem361_k_eigenvalueand -
smrg7
cc: @braxtoncuneo, @jpmorgan98