Did the weight_decay needed?

[wzn0828]

Hi, thanks for your wanderful work.

As I use your AdaptiveLossFunction, I found the alpha did not decrease, it keeps the highest value through the training process.

So, I used the weight_decay to the alpha and the scale. However, I think the weight_decay should not be used for the two parameters.

What' your opinion?

[jonbarron]

If the alpha value stays large throughout optimization, it sounds like your data doesn't have very many outliers, in which case you'll probably get optimal performance by just allowing alpha to be large. Regularizing alpha to be small does not make much sense to me unless you have a prior belief on the outlier distribution of your data. If you want to control the shape of the loss function, I'd just use the general formulation in general.py, and set alpha to whatever value you want.

[wzn0828]

Ok, your answer addresses my mystery. Thank you very much.