visualizing_momentum

Visualizing Gradient Descent with Momentum in Python, check out the blog post here!

Pre-reqs

Python 3.6

Libraries

• matplotlib
• numpy

How to run

python3 loss_surface.py to generate loss surface figure.

python3 momentum.py to generate weight trajectories and velocity-iteration plots.

References

• An overview of gradient descent algorithms
• On the importance of initialization and momentum in deep learning
• Why momentum really works
• Momentum and learning rate adaptation