nni
nni copied to clipboard
Enhancement of one-shot NAS (v2.9)
Description
Features
- Add export_probs to monitor the architecture weights.
- Rewrite configure_optimizers, functions to step optimzers / schedulers, along with other hooks for simplicity, and to be compatible with latest lightning (v1.7).
- Support gradient clipping in DARTS.
- Support arbitary arguments in random one-shot, because it's automatic optimization.
- Support +1 (none) in differentiable cell, to mimic the input choice (similar to what DARTS did).
- Reimplement ProxylessNAS with forward/backward hooks. This increases its robustness and applicability.
Bug fixes
- Fix a misuse of lambda expression in NAS-Bench-201 (used in tests).
- Fix the gumbel temperature (the original one doesn't work at all).
- Fix an issue that formats tensors in
weighted_sum
, making it very slow. - Fix the label sharing in differentiable strategies. The original one wasn't sharing at all.
- Fix the memo reuse in exporting differentiable cell.
Misc
- Add a more friendly error message when no architecture weights are available.
- Added some more explanations in one-shot documentation: manual optimization, some extra details in one-shot supermodules, typo fixes.
Test Options
- [ ] fast test
- [ ] full test - HPO
- [ ] full test - NAS
- [ ] full test - compression
Checklist
- [ ] test case
- [ ] doc
How to test
In another PR.