math-and-ml-notes

math-and-ml-notes copied to clipboard

Links to some important research papers or links. I plan to add notes as I go through each topic one by one.

✔ 1. Information theory based (unsupervised) learning

• [x] Invariant Information Clustering
  • Notes
• [x] Mutual Information Neural Estimation
  • Notes
• [x] Deep Infomax
  • Notes
• [x] Learning Representations by Maximizing Mutual Information Across Views
  • Notes
• [x] How Google decoupled MI maximization and representation learning: On Mutual Information Maximization for Representation Learning
  • Notes

✔ 2. Disentangled representations

• [x] Quick overview by Google
  • Notes
• [x] β-VAE, pdf
  • Notes
• [x] Understanding disentangling in β-VAE
  • Notes

Decided not to delve deeper into this topic. It is not mature yet.
* [Disentangling Disentanglement in Variational Autoencoders](https://arxiv.org/abs/1812.02833)
* [Isolating Sources of Disentanglement in Variational Autoencoders](https://arxiv.org/abs/1802.04942)
* [InfoGAN-CR: Disentangling Generative Adversarial Networks with Contrastive Regularizers](https://arxiv.org/abs/1906.06034)
* [Disentangling by Factorising, pdf](https://www.cs.toronto.edu/~amnih/papers/disentangling_nips_ws.pdf)

✔ 3. Self Supervised Learning

• [x] Representation Learning with Contrastive Predictive Coding
• [x] Data-Efficient Image Recognition with Contrastive Predictive Coding
  • Combined notes
• [x] Contrastive Multiview Coding
  • Notes
• [x] Momentum Contrast for Unsupervised Visual Representation Learning
  • Notes
• [x] SimCLR
  • Notes
• [x] MoCo v2: Improved Baselines with Momentum Contrastive Learning
  • The shortest paper I have seen. The authors modified MoCo based on some ideas borrowed from SimCLR.
  • Notes
• [ ] Emerging Properties in Self-Supervised Vision Transformers PDF | Official Code

✔ 4. Automatic differentiation

• [x] Automatic differentiation in machine learning: a survey
  • Annotated pdf
• [x] Automatic Reverse-Mode Differentiation: Lecture Notes
  • Annotated pdf
• [x] Reverse mode automatic differentiation

5. NNs and ODEs

• [ ] Neural Ordinary Differential Equations
  • [ ] Adjoint tutorial
• [ ] Augmented Neural ODEs
• [ ] Invertible ResNets
• [ ] Universal Differential Equations for Scientific Machine Learning

6. Probabilistic Programming

• [x] Probabilistic models of cognition
  • Notes
• [ ] The Design and Implementation of Probabilistic Programming Languages
• [ ] Composition in Probabilistic Language Understanding

7. Miscellaneous

Memorization in neural networks

• Blog post by BAIR
• Identity Crisis: Paper

Online Learning

• A Modern Introduction to Online Learning

Graph Neural Networks

• [ ] A Gentle Introduction to Deep Learning for Graphs, pdf

Normalizing Flows

• [x] Tutorial and implementations for UCB DUL course

Transformers

• [x] Attention is all you need
  • [x] A nice visual introduction
  • [x] Annonated paper
  • Python notebook with illustrations and working code
• Reformer: The Efficient Transformer

Others

• Zero-shot knowledge transfer
• SpecNet
• Deep Learning & Symbolic Mathematics
• Deep Equilibrium Models

8. Theory of neural networks

Lottery tickets

• Lottery ticket hypothesis
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Rigging the Lottery: Making All Tickets Winners

Others

• What's Hidden in a Randomly Weighted Neural Network?
• Topological properties of the set of functions generated by neural networks of fixed size
• YOUR CLASSIFIER IS SECRETLY AN ENERGY BASED MODEL AND YOU SHOULD TREAT IT LIKE ONE
• Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology

9. Advanced Variational Inference

• Amortized Population Gibbs Samplers with Neural Sufficient Statistics
• Evaluating Combinatorial Generalization in Variational Autoencoders

10. Causal Inference

• Elements of Causal Inference, pdf
• Causal Inference: What If, pdf
• Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution