physicsnemo icon indicating copy to clipboard operation
physicsnemo copied to clipboard
Published 3 days ago • verified publisher icon NVIDIA

⛰️[EPIC]: CorrDiff Usability Enhancements

Open mnabian opened this issue 1 year ago • 2 comments 

### Tasks - P0
- [ ] https://github.com/NVIDIA/modulus/issues/639
- [ ] https://github.com/NVIDIA/modulus/issues/633
- [ ] https://github.com/NVIDIA/modulus/issues/632
- [ ] https://github.com/NVIDIA/modulus/issues/614
- [ ] https://github.com/NVIDIA/modulus/issues/613
- [ ] CorrDiff: Generalized dataloader
- [ ] https://github.com/NVIDIA/modulus/issues/612
- [ ] CorDiff: Unit tests for training/inference, put it in the examples folder
- [ ] https://github.com/NVIDIA/modulus/issues/601
- [ ] https://github.com/NVIDIA/modulus/issues/611
- [ ] https://github.com/NVIDIA/modulus/issues/600
- [ ] https://github.com/NVIDIA/modulus/issues/599
- [ ] https://github.com/NVIDIA/modulus/issues/634
- [ ] https://github.com/NVIDIA/modulus/pull/594
- [ ] https://github.com/NVIDIA/modulus/issues/541
- [ ] https://github.com/NVIDIA/modulus/issues/537
- [ ] https://github.com/NVIDIA/modulus/issues/313
- [ ] https://github.com/NVIDIA/modulus/issues/542
- [ ] https://github.com/NVIDIA/modulus/issues/415
- [ ] https://github.com/NVIDIA/modulus/issues/655

### Tasks - P1
- [x] https://github.com/NVIDIA/modulus/issues/598
- [ ] CorrDiff: Support distributing generation over time steps as well as ensemble members
- [ ] https://github.com/NVIDIA/modulus/issues/602
- [ ] https://github.com/NVIDIA/modulus/issues/635
- [ ] https://github.com/NVIDIA/modulus/issues/623
- [ ] CorrDiff: Avoid hard-coding the validation year in the datapipe
- [ ] https://github.com/NVIDIA/modulus/issues/622
- [ ] https://github.com/NVIDIA/modulus/issues/478

mnabian avatar Jul 12 '24 17:07 mnabian

@mnabian So looking through the task list, we are rearchitecting the code to make it cleaner. From a user perspective, what will they be able to do more easily in terms of say what types of data formats can they ingest, does our training code provide configs that make it easy to train with the supported data formats?

ram-cherukuri avatar Aug 02 '24 17:08 ram-cherukuri

Scope of the enhancements:

  • Re-architect Training and Generation: Significantly refactor training and generation processes to hide unnecessary details and configurations, improving modularity, and removing non-intuitive abstractions for a more user-friendly experience.

  • Enhance Configuration Management: Improve the handling of configurations for more efficient, generalizable, and user-friendly setup. Also hide unnecessary configurations from the users to improve usability.

  • Simplify Metrics Logging: Streamline the logging process for training and validation metrics to provide clear and accessible performance tracking.

  • Develop CorrDiff-Lite: Create a simplified, educational version of CorrDiff training and generation, demonstrated on the HRRR dataset, to facilitate learning and accessibility.

  • Refactor and Generalize Dataloader: Enhance and generalize the dataloader to be served as a template for the users to implement their own customized dataloaders.

  • Implement Regression Tests: Introduce regression tests for CorrDiff training and generation to ensure accuracy and reliability in future code updates.

  • Develop Comprehensive Documentation: Create thorough and user-friendly documentation to support all aspects of CorrDiff, ensuring users have clear guidance and resources. This includes both README documentation as well as API documentation.

mnabian avatar Aug 06 '24 18:08 mnabian

Closing as complete.

https://github.com/NVIDIA/modulus/pull/604 https://github.com/NVIDIA/modulus/pull/629 https://github.com/NVIDIA/modulus/pull/636 https://github.com/NVIDIA/modulus/pull/676

mnabian avatar Oct 17 '24 22:10 mnabian