Draft: Non-regression tests for CorrDiff training and generation

[CharlelieLrt]

PhysicsNeMo Pull Request

Description

To improve stability of CorrDiffr, this PR introduces non-regression for the end-to-end workflow of CorrDiff train.py and generate.py. The goal is to have non-regression tests that combine:

• lead-time aware models
• regression and diffusion architectures representative of an actual CorrDiff application
• patching
• latest performance optimizations (AMP< compiling, Apex GN, etc...)

More specifically, the PR introduces:

• Regression and diffusion model checkpoints, generated with release v1.0.1
• Reference loss data for CorrDiff training, generated with release v1.1.1
• Non-regression test for CorrDiff regression/diffusion training, which ensures that results from current model implementation do not deviate from that obtained with checkpoints from v1.0.1
• Non-regression tests for CorrDiff generation using both deterministic/stochastic sampler with model checkpoints and current APIs

Checklist

• [x] I am familiar with the Contributing Guidelines.
• [x] New or existing tests cover these changes.
• [x] The documentation is up to date with these changes.
• [x] The CHANGELOG.md is up to date with these changes.
• [ ] An issue is linked to this pull request.

Dependencies