optimizer-visualization

Visualize gradient descent optimization algorithms in Tensorflow.

All methods start at the same location, specified by two variables. Both x and y variables are improved by the following Optimizers:

Adadelta documentation

Adagrad documentation

Adam documentation

Ftrl documentation

GD documentation

Momentum documentation

RMSProp documentation

For an overview of each gradient descent optimization algorithms, visit this helpful resource.

Numbers in figure legend indicate learning rate, specific to each Optimizer.

Note the optimizers' behavior when gradient is steep.

Note the optimizers' behavior when initial gradient is miniscule.

Inspired by the following GIFs:

From here