nni icon indicating copy to clipboard operation
nni copied to clipboard

Enhancement of one-shot NAS (v2.9)

Open ultmaster opened this issue 2 years ago • 0 comments

Description

Features

  • Add export_probs to monitor the architecture weights.
  • Rewrite configure_optimizers, functions to step optimzers / schedulers, along with other hooks for simplicity, and to be compatible with latest lightning (v1.7).
  • Support gradient clipping in DARTS.
  • Support arbitary arguments in random one-shot, because it's automatic optimization.
  • Support +1 (none) in differentiable cell, to mimic the input choice (similar to what DARTS did).
  • Reimplement ProxylessNAS with forward/backward hooks. This increases its robustness and applicability.

Bug fixes

  • Fix a misuse of lambda expression in NAS-Bench-201 (used in tests).
  • Fix the gumbel temperature (the original one doesn't work at all).
  • Fix an issue that formats tensors in weighted_sum, making it very slow.
  • Fix the label sharing in differentiable strategies. The original one wasn't sharing at all.
  • Fix the memo reuse in exporting differentiable cell.

Misc

  • Add a more friendly error message when no architecture weights are available.
  • Added some more explanations in one-shot documentation: manual optimization, some extra details in one-shot supermodules, typo fixes.

Test Options

  • [ ] fast test
  • [ ] full test - HPO
  • [ ] full test - NAS
  • [ ] full test - compression

Checklist

  • [ ] test case
  • [ ] doc

How to test

In another PR.

ultmaster avatar Aug 08 '22 00:08 ultmaster