Add more optimizers and losses

[Shashi456]

Just a small-ish roadmap to different Optimizers and losses we can look at to add :

Optimizers:-

• [x] Adam
• [x] Adagrad
• [x] SGD
• [x] RMSprop
• [x] AdaDelta ~- [x] Riemann SGD~ Removed by #311
• [ ] Adabound ->ICLR 2019 -> Repo
• [ ] LBFGS
• [x] AdaMax
• [ ] SparseAdam

Losses :-

• [x] L1Loss
• [x] L2Loss
• [x] MeanSquaredError
• [x] SoftmaxCrossEntropy
• [x] SoftmaxCrossEntropyWithLogits
• [x] SigmoidCrossEntropy
• [x] MeanAbsoluteError
• [x] MeanAbsolutePercentageError
• [x] MeanSquaredLogarithmicError
• [x] HingeLoss
• [x] SquaredHinge
• [x] CategoricalHinge
• [x] Logcosh
• [x] CategoricalCrossEntropy
• [x] Kullback Leibler Divergence ~NegativeLogLikelihood~
• [x] Cosine
• [ ] TripletMarginLoss
• [x] Poisson

Removed NegativeLogLikelihood. Check this discussion for more.

Note: If you have suggestions for losses and optimizers, please suggest them in this issue.

[rxwei]

Thanks for listing these!

[Shashi456]

@rxwei just a small roadmap, so that we have in mind what we need to add.

[liberium]

Maybe it would be better to organise this as a project?

[Shashi456]

@liberium. This is not priority right now. I wrote this task list so that we know what we have and what we don't. We are currently implementing all the layers. This is just a simple checklist so that we know where we are, and if someone wants to implement they can help us. Issues are a good way to inform people about open source contributions. In this case, a "good first issue"

[Shashi456]

@dan-zheng @rxwei so this Google group thread here talks about regularization. Although we are a little far from that stage right now.

One thing I was going to suggest him to look at was to retrieve the layer.weight iteratively, but then realized a lot of layers don't have the weight parameter.

I think one functionality we should look at adding is retrieving model weights. Is there an easy/hack-y way to add this ? Could we start looking at this I would definitely like to help in adding this functionality.

[eaplatanios]

@Shashi456 I believe that regularization schemes are heavily intertwined with the training abstractions, which do not yet exist in this repo. It would be nice to start a discussion around them at some point though.

[jon-tow]

@Shashi456 Categorical Cross-Entropy seems to already be implemented through Softmax Cross-Entropy with Logits. Maybe we can cross it off the list?

[Shashi456]

@rxwei So I'm almost done with adding all the losses. Thanks to @Descartess and @jon-tow for helping out in that matter. One of the only listed losses remaining is #31 which is the triplet loss, If you merge it I can add a test for it.

cc : @dan-zheng @eaplatanios

[Shashi456]

@eaplatanios is there any optimizer which hasn't been refactored yet?

[eaplatanios]

All that can be refactored currently, have been. We are missing some functionality that would allow us to simplify AdaMax that is not straightforward to add so that’s the only one using key paths right now.

[iyaja]

Have you considered adding the LAMB optimizer to the list? It seems to be one of the fastest so far (especially with large batches).

[jph00]

FYI there's a good set of optimizers and a nice simple API for adding new ones in SwiftAI, including LAMB. It might be better to contribute to SwiftAI if you've got additional optimizers to add.

[Shashi456]

@jph00, thanks :). Will start contributing some optimizers there as well.

[vballoli]

#564 Add Rectified Adam Optimizer

[Kshitij09]

LabelSmoothingCrossEntropy Loss, originally proposed by Szegedy et al. and further expounded by Geoffrey Hinton et al. in When does Label Smoothing Help? have been proven effective strategy for building robust models. While it's relatively simple to integrate, I'd like to contribute this one to get acquainted with swift for tensorflow! cc: @Shashi456 could you put this one in the list?

[Kshitij09]

I've few things to discuss regarding the current implementation of softmaxCrossEntropy:

1. I found inconsistencies among docs of tensorflow's sparse_softmax_cross_entropy and swift-api's softmaxCrossEntropy(logits:labels:reduction). The reason for this comparison is the effective call being made by this function.

As per tensorflow's doc, logits are unscaled log probabilites whereas swift-apis say they're One-hot encoded outputs from a neural network. Also, there's one warning in tensorflow docs WARNING: This op expects unscaled logits, since it performs a softmax on logits internally for efficiency. Do not call this op with the output of softmax, as it will produce incorrect results.

1. As it is mentioned in the docs of softmaxCrossEntropy, "Each row must be valid probability distribution", however, there's no such precondition in the current implementation.

2. Adhering to tf.compat.v1 implementation of CategoricalCrossentropy loss

tf.losses.softmax_cross_entropy(
    onehot_labels,
    logits,
    weights=1.0,
    label_smoothing=0,
    scope=None,
    loss_collection=tf.GraphKeys.LOSSES,
    reduction=Reduction.SUM_BY_NONZERO_WEIGHTS
)

We could extend current implementation of softmaxCrossEntropy with label_smoothing and weights, allowing us to create both Weighted as well as LabelSmoothingCrossEntropy. I'd like to make above mentioned changes and seeking for your thoughts on the same. I've done some experiments with raw and one-hot encoded inputs and I found that loss varies drastically.

cc: @dan-zheng @eaplatanios