Implement orthogonal weight initialisation

[mratsim]

Seems to be key for RNNs to learn long-term dependencies.

Foundational paper: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks 2013, Saxe et al, https://arxiv.org/abs/1312.6120

Papers

• Neural Photo Editing with Introspective Adversarial Networks, Brock et al, https://arxiv.org/abs/1609.07093
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs, Jing et al (Lecun supervision), https://arxiv.org/abs/1612.05231
• On orthogonality and learning recurrent networks with long term dependencies, Vorontsov et al, https://arxiv.org/abs/1702.00071
• Recurrent Orthogonal Networks and Long-Memory Tasks, Henaff et al (Lecun supervision), https://arxiv.org/abs/1602.06662
• Regularizing RNNs by Stabilizing Activations, Krueger et al, https://arxiv.org/abs/1511.08400
• On the difficulty of training Recurrent Neural Networks, Pascanu et al (Bengio supervision), https://arxiv.org/abs/1211.5063
• Initialization Matters: Orthogonal Predictive State Recurrent Neural Networks, Choromanski et al, https://openreview.net/pdf?id=HJJ23bW0b