addons

addons copied to clipboard

Describe the feature and the current behavior/state.

• GRU cell (RNNs) with layernorm layers.
• Similar to LayerNormLSTMCell https://github.com/tensorflow/addons/blob/d126b4690e7858b85fdb3c9e9a61143c31e9fddb/tensorflow_addons/rnn/layer_norm_lstm_cell.py

Relevant information

• Are you willing to contribute it (yes/no): no (too busy rn, sorry)
• Are you willing to maintain it going forward? (yes/no): maybe
• Is there a relevant academic paper? (if so, where):
  • Ba et al (2016) arXiv:1607.06450 for layernorm itself.
  • Wei (2021), "Distantly Supervision for Relation Extraction via LayerNorm Gated Recurrent Neural Networks", IEEE https://doi.org/10.1109/CDS52072.2021.00022
  • Zhang/Sennrich (2019) propose RMSNorm instead of LayerNorm and test it with GRU, https://proceedings.neurips.cc/paper/2019/hash/1e8a19426224ca89e83cef47f1e7f53b-Abstract.html
• Does the relavent academic paper exceed 50 citations? (yes/no): only Ba et al (2016)
• Is there already an implementation in another framework? (if so, where): Yes, https://github.com/ElektrischesSchaf/LayerNorm_GRU in PyTorch by @ElektrischesSchaf
• Was it part of tf.contrib? (if so, where): idk

Which API type would this fall under (layer, metric, optimizer, etc.) layer, rnn

Who will benefit with this feature? @thegodone requested the feature in dev code repo for the SimpleRNNCell (https://github.com/kmedian/keras-layernorm-rnn/issues/2). However, TFA is the appropriate place for such request.

Any other info.

yes that would be cool to have LayerNormGRU version in addons

any update?

TensorFlow Addons is transitioning to a minimal maintenance and release mode. New features will not be added to this repository. For more information, please see our public messaging on this decision: TensorFlow Addons Wind Down

Please consider sending feature requests / contributions to other repositories in the TF community with a similar charters to TFA: Keras Keras-CV Keras-NLP