hfta

hfta copied to clipboard

Type error with lr_scheduler

1

After several epochs running, an error will raise as following: ``` /hfta/optim/lr_scheduler.py", line 247, in _update_lr res = this_lr * multiplier TypeError: unsupported operand type(s) for *: 'Coefficient' and 'float'...

zyfhaha

* [y ] Tested * [y ] Formatted with YAPF

xinli-git

Ops: - [x] batchnorm.py - [x] conv.py - [x] dropout2d.py - [x] embedding.py - [x] layernorm.py - [x] linear.py - [ ] [no change needed?] multiheadattention.py - [x] pool.py -...

wangshangsam

[Optim] Upgrade adam to 1.9

3

* [x] Tested * [x] Formatted with YAPF PyTorch 1.9 uses a functional API to perform Adam computation. However, it seems like we don't have a functional API for those...

eric-zheng

Triggering the test runs on each PR.

wangshangsam

- [ ] Convergence Testing: Using the existing examples. Comparing between HFTA vs. serial. Each with 5 runs, one seed per run, the compare the mean and std of the...

wangshangsam

wangshangsam

PyTorch native AMP (i.e., `torch.cuda.amp`) interacts with the optimizers and expects certain behaviors out of the optimizers. We should add unit tests to make sure that those interactions are fine.

wangshangsam