About `NoamLR`

[ShaneTian]

https://github.com/tugstugi/pytorch-saltnet/blob/a3e63b357f975924e75b5db24ee528c5797f5efb/utils/lr_scheduler.py#L36-L39

The custom NoamLR results in same LR in the first two steps, like:

warmup_steps = 10
lr = 0.01

• step -- before (current step LR) -- after (next step LR)
• 0 -- 0.001 -- 0.001
• 1 -- 0.001 -- 0.002
• 2 -- 0.002 -- 0.003

---

There are two ways to fix this:

• last_epoch = self.last_epoch + 1, like ESPnet

  def get_lr(self): 
      last_epoch = self.last_epoch + 1
      scale = self.warmup_steps ** 0.5 * min(last_epoch ** (-0.5), last_epoch * self.warmup_steps ** (-1.5)) 
      return [base_lr * scale for base_lr in self.base_lrs] 
  

• use LambdaLR directly

  noam_scale = lambda epoch: (warmup_steps ** 0.5) * min((epoch + 1) ** -0.5, (epoch + 1) * (warmup_steps ** -1.5))
  scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=noam_scale)
  

Of course, the above two approaches are equivalent.