lr scheduling

[egozi]

Is your feature request related to a problem? Please describe. Add more learning rate scheduling options

Describe the solution you'd like Include a flag/operation to choose other lr scheduling than reduce-on-plateau. In particular, cosine-annealing scheduling is very popular and should be part of the package.

Describe alternatives you've considered NA

Additional context NA

[mosheraboh]

@egozi, you should be able to directly use any implemented PyTorch lr sch in here https://pytorch.org/docs/stable/optim.html - it includes cosine-annealing. Did you have an issue with it?