Implement Recurrent Layers

[adam-r-kowalski]

It would be really nice to have a set of recurrent layers including LSTMs and GRUs as part of the core layers API. Ideally we can parameterize the activation functions for the various gates rather than hard coding in the tanh and sigmoid functions.

[rxwei]

Definitely, they are on our roadmap. Thanks for opening an issue about this. And, contributions are welcome!

[rxwei]

FWIW, here's an example RNN implemented using the Swift of TensorFlow Deep Learning Library: https://gist.github.com/rxwei/ce6644efad8f229651050e096c05ccbb.

Most people are often stuck at these two things on their first try:

• Defining a layer with multiple inputs and outputs.
  • Answer: Define a struct Input : Differentiable { ... } and struct Output : Differentiable { ... } in your layer with however many arity you need. As to variable-sized inputs/outputs, Array will conform to Differentiable soon.
• Backpropagation through time without differentiating control flow.
  • Answer: Control flow differentiation is not required for BPTT. Each Layer has a Backpropagator member type. Use appliedForBackpropagation to evaluate the forward pass and get a back propagator, and accumulate backpropagators in an array.

You can find how these are done in my gist. If anyone's interested in taking a stab to implement RNN cells, let me know!

[rxwei]

Update: #80 added the RNNCell protocol and #71 added SimpleRNN and LSTM. We still need to add GRU and hopefully NTM.

[dhasl002]

I'd like to contribute. I'll work on GRU and NTM

[dhasl002]

Update: #417 has created to implement GRU cells. I intend to move on to NTM next.