Awesome Memory in RL

A curated list of conference papers studying memory mechanisms for reinforcement learning. Also check awesome-offline-rl, awesome-ebm, awesome-model-mbrl. Forks and PRs are welcome.

Reinforcement Learning

2021

• End-to-End Egospheric Spatial Memory
  • Daniel Lenton, Stephen James, Ronald Clark, Andrew J. Davison [ICLR]
• Learning Associative Inference Using Fast Weight Memory
  • Imanol Schlag, Tsendsuren Munkhdalai, Jürgen Schmidhuber [ICLR]
• Solving Continuous Control with Episodic Memory
  • Igor Kuznetsov, Andrey Filchenkov [IJCAI]
• Generalizable Episodic Memory for Deep Reinforcement Learning
  • Hao Hu, Jianing Ye, Guangxiang Zhu, Zhizhou Ren, Chongjie Zhang [ICML]

2020

• Episodic Reinforcement Learning with Associative Memory
  • Guangxiang Zhu, Zichuan Lin, Guangwen Yang, Chongjie Zhang [ICLR]
• AMRL: Aggregated Memory For Reinforcement Learning
  • Jacob Beck, Kamil Ciosek, Sam Devlin, Sebastian Tschiatschek, Cheng Zhang, Katja Hofmann [ICLR]
• Sparse Graphical Memory for Robust Planning
  • Scott Emmons, Ajay Jain, Michael Laskin, Thanard Kurutach, Pieter Abbeel, Deepak Pathak [NeurIPS]
• Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewards
  • Yijie Guo, Jongwook Choi, Marcin Moczulski, Shengyu Feng, Samy Bengio, Mohammad Norouzi, Honglak Lee [NeurIPS]
• Working Memory Graphs
  • Ricky Loynd, Roland Fernandez, Asli Celikyilmaz, Adith Swaminathan, Matthew Hausknecht [ICML]
• Hallucinative Topological Memory for Zero-Shot Visual Planning
  • Kara Liu, Thanard Kurutach, Christine Tung, Pieter Abbeel, Aviv Tamar [ICML]

2019

• Episodic Curiosity through Reachability
  • Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, Sylvain Gelly [ICLR]
• Generalization of Reinforcement Learners with Working and Episodic Memory
  • Meire Fortunato, Melissa Tan, Ryan Faulknel et. al [NeurIPS]
• Policy Consolidation for Continual Reinforcement Learning
  • Christos Kaplanis, Murray Shanahan, Claudia Clopath [ICML]
• Remember and Forget for Experience Replay
  • Guido Novati, Petros Koumoutsakos [ICML]
• Reinforcement Learning, Fast and Slow
  • Matthew Botvinick, Sam Ritter, Jane X. Wang, Zeb Kurth-Nelson, Charles Blundell et. al [Trends in Cognitive Sciences]

2018

• Memory Augmented Control Networks
  • Arbaaz Khan, Clark Zhang, Nikolay Atanasov, Konstantinos Karydis, Vijay Kumar, Daniel D. Lee [ICLR]
• Neural Map: Structured Memory for Deep Reinforcement Learning
  • Emilio Parisotto, Ruslan Salakhutdinov [ICLR]
• Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
  • Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc Le, Ni Lao [NeurIPS]
• Fast deep reinforcement learning using online adjustments from the past
  • Steven Hansen, Pablo Sprechmann, Alexander Pritzel, André Barreto, Charles Blundell [NeurIPS]
• Continual Reinforcement Learning with Complex Synapses
  • Christos Kaplanis, Murray Shanahan, Claudia Clopath [ICML]
• Been There, Done That: Meta-Learning with Episodic Recall
  • Samuel Ritter, Jane X. Wang, Zeb Kurth-Nelson, Siddhant M. Jayakumar, Charles Blundell et. al [ICML]
• Episodic Memory Deep Q-Networks
  • Zichuan Lin, Tianqi Zhao, Guangwen Yang, Lintao Zhang [IJCAI]
• Unsupervised Predictive Memory in a Goal-Directed Agent
  • Greg Wayne, Chia-Chun Hung, David Amos et. al

2017

• Fast Reinforcement Learning via Slow Reinforcement Learning
  • Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel [ICLR]
• Neural Episodic Control
  • Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech, Oriol Vinyals, Demis Hassabil et. al [ICML]

... - 2016

• Using Fast Weights to Attend to the Recent Past
  • Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu [NIPS-2016]
• Control of Memory, Active Perception, and Action in Minecraft
  • Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, Honglak Lee [ICLR-2016]
• Model-Free Episodic Control
  • Charles Blundell, Benigno Uria, Alexander Pritzel et. al
• Hippocampal Contributions to Control: The Third Way
  • Máté Lengyel, Peter Dayan [NIPS-2007]

Deep Learning

2021

• Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting
  • Sayna Ebrahimi, Suzanne Petryk, Akash Gokul, William Gan, Joseph E. Gonzalez, Marcus Rohrbach, Trevor Darrell [ICLR]
• Gradient Projection Memory for Continual Learning
  • Gobinda Saha, Isha Garg, Kaushik Roy [ICLR]
• Learn from Concepts: Towards the Purified Memory for Few-shot Learning
  • Xuncheng Liu, Xudong Tian, Shaohui Lin, Yanyun Qu, Lizhuang Ma, Wang Yuan, Zhizhong Zhang, Yuan Xi [IJCAI]
• Not All Memories are Created Equal: Learning to Forget by Expiring
  • Sainbayar Sukhbaatar, Da Ju, Spencer Poff, Stephen Roller, Arthur Szlam, Jason Weston, Angela Fan [ICML]

2020

• Memory-Based Graph Networks
  • Amir Hosein Khasahmadi, Kaveh Hassani, Parsa Moradi, Leo Lee, Quaid Morris [ICLR]
• Meta-Learning Deep Energy-Based Memory Models
  • Sergey Bartunov, Jack W Rae, Simon Osindero, Timothy P Lillicrap [ICLR]
• MEMO: A Deep Network for Flexible Combination of Episodic Memories
  • Andrea Banino, Adrià Puigdomènech Badia, Raphael Köster et. al [ICLR]
• Progressive Memory Banks for Incremental Domain Adaptation
  • Nabiha Asghar, Lili Mou, Kira A. Selby, Kevin D. Pantasdo, Pascal Poupart, Xin Jiang [ICLR]
• Neural Stored-program Memory
  • Hung Le, Truyen Tran, Svetha Venkatesh [ICLR]
• H-Mem: Harnessing synaptic plasticity with Hebbian Memory Networks
  • Thomas Limbacher and Robert Legenstein [NeurIPS]
• Online Multitask Learning with Long-Term Memory
  • Mark Herbster, Stephen Pasteris, Lisa Tse [NeurIPS]
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
  • Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, Christopher Re [NeurIPS]
• Learning to Learn Variational Semantic Memory
  • Xiantong Zhen, Yingjun Du, Huan Xiong, Qiang Qiu, Cees G. M. Snoek, Ling Shao [NeurIPS]
• Improved Schemes for Episodic Memory-based Lifelong Learning
  • Yunhui Guo, Mingrui Liu, Tianbao Yang, Tajana Rosing [NeurIPS]
• Self-Attentive Associative Memory
  • Hung Le, Truyen Tran, Svetha Venkatesh [ICML]
• Associative Memory in Iterated Overparameterized Sigmoid Autoencoders
  • Yibo Jiang, Cengiz Pehlevan [ICML]
• Multigrid Neural Memory
  • Tri Huynh, Michael Maire, Matthew R. Walter [ICML]

2019

• Learning to Remember More with Less Memorization
  • Hung Le, Truyen Tran, Svetha Venkatesh [ICLR]
• Adaptive Posterior Learning: few-shot learning with a surprise-based memory module
  • Tiago Ramalho, Marta Garnelo [ICLR]
• Large Memory Layers with Product Keys
  • Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou [NeurIPS]
• Episodic Memory in Lifelong Language Learning
  • Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, Dani Yogatama [NeurIPS]
• Metalearned Neural Memory
  • Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, Adam Trischler [NeurIPS]
• Ordered Memory
  • Yikang Shen, Shawn Tan, Arian Hosseini, Zhouhan Lin, Alessandro Sordoni, Aaron Courville [NeurIPS]
• Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks
  • Aaron R. Voelker, Ivana Kajic ́, Chris Eliasmith [NeurIPS]

2018

• Semi-parametric Topological Memory for Navigation
  • Nikolay Savinov, Alexey Dosovitskiy, Vladlen Koltun [ICLR]
• Memory-based Parameter Adaptation
  • Pablo Sprechmann, Siddhant M. Jayakumar, Jack W. Rae, Alexander Pritzel et. al [ICLR]
• Convolutional Memory Blocks for Depth Data Representation Learning
  • Keze Wang, Liang Lin, Chuangjie Ren, Wei Zhang, Wenxiu Sun [IJCAI]
• Visual Memory for Robust Path Following
  • Ashish Kumar, Saurabh Gupta, David Fouhey, Sergey Levine, Jitendra Malik [NeurIPS]
• A Simple Cache Model for Image Recognition
  • A. Emin Orhan [NeurIPS]
• Variational Memory Encoder-Decoder
  • Hung Le, Truyen Tran, Thin Nguyen, Svetha Venkatesh [NeurIPS]
• Fast Parametric Learning with Activation Memorization
  • Jack W Rae, Chris Dyer, Peter Dayan, Timothy P Lillicrap [ICML]
• Learning and Memorization
  • Satrajit Chatterjee [ICML]

2017

• Reasoning with Memory Augmented Neural Networks for Language Comprehension
  • Tsendsuren Munkhdalai, Hong Yu [ICLR]
• Learning to Remember Rare Events
  • Łukasz Kaiser, Ofir Nachum, Aurko Roy, Samy Bengio [ICLR]
• Variational Memory Addressing in Generative Models
  • Jörg Bornschein, Andriy Mnih, Daniel Zoran, Danilo J. Rezende [NIPS]
• A simple model of recognition and recall memory
  • Nisheeth Srivastava, Edward Vul [NIPS]
• Gradient Episodic Memory for Continual Learning
  • David Lopez-Paz, Marc'Aurelio Ranzato [NIPS]

... - 2016

• End-To-End Memory Networks [NIPS-205]

Cognition and Neuroscience

• Large Associative Memory Problem in Neurobiology and Machine Learning
  • Dmitry Krotov, John Hopfield [ICLR-2021]
• Compositional Explanations of Neurons
  • Jesse Mu, Jacob Andreas [NeurIPS-2020]
• Coordinated hippocampal-entorhinal replay as structural inference
  • Talfan Evans, Neil Burgess [NeurIPS-2019]
• Generalisation of structural knowledge in the hippocampal-entorhinal system
  • James C. R. Whittington, Timothy H. Muller, Shirley Mark, Caswell Barry, Timothy E. J. Behrens [NeurIPS-2018]
• Dendritic cortical microcircuits approximate the backpropagation algorithm
  • João Sacramento, Rui Ponte Costa, Yoshua Bengio, Walter Senn [NeurIPS-2018]