Timely and Differential Dataflow 101

This a collection of resources for people interested in learning more about Timely Dataflow and Differential Dataflow.

F.A.Q.

What's the difference between Timely and Differential Dataflow?

TODO

Resources

• [ ] 📖 Materialize Tech Blog
  • [ ] 📖 Building Differential Dataflow from Scratch (2023) by Ruchir Khaitan
  • [ ] 📖 Life in Differential Dataflow (2021) by Ruchir Khaitan (introduction with examples)
• [ ] 📕 Timely Dataflow book (2017–) by Frank McSherry (intro and internals)
• [ ] 📕 Differential Dataflow book (2018–) by Frank McSherry (basic and advanced usage)
• [ ] 👩🏽‍💻 Timely Lab by Moritz Hoffmann (a hands-on introduction; slides)
• [ ] 📖 Blog by Frank McSherry (some selected posts below; many more available)
  • [ ] Generalizing Linear Operators (2021)
  • [ ] Rust for Data-Intensive Computation (2020)
  • [ ] Differential Dataflow meets calculus (2020)
  • [ ] Tracking Progress in Timely Dataflow (2019)
  • [ ] Timely Dataflow Architecture (2018)
  • [ ] Making Arrangements (2017)
  • [ ] Dataflow as Database (2016)
  • [ ] An Introduction to Differential Dataflow (Part 1, 2) (2015)
  • [ ] An Introduction to Timely Dataflow in Rust (Part 1, 2, 3) (2015)
  • [ ] Scalability! But at what COST? (2014)
• [ ] 🎥 YouTube channel (2019–) by Frank McSherry (mostly live coding; some conceptual explanations)
• [ ] 📖 Blog by Justin Jaffray
  • [ ] Compaction (2021)
  • [ ] Timely Dataflow and Total Order (2020)
• [ ] 📖 Blog by Jamie Brandon
  • [ ] An opinionated map of incremental and streaming systems (2021)
  • [ ] Internal consistency in streaming systems (2021)
  • [ ] Why isn't differential dataflow more popular? (2021)
• [ ] 🎓 Naiad: a Timely Dataflow System (2013) by Murray et al. (research paper on Timely's precursor at Microsoft Research)
• [ ] 🎓 Differential Dataflow (2013) by McSherry et al. (research paper on Differential Dataflow at Microsoft Research)
• [ ] 🎓 Shared Arrangements: Practical Inter-query Sharing for Streaming Dataflows (2020) by McSherry et al. (research paper on the arrangement index structure)
• [ ] 🎙 Real-time Data Streaming with Frank McSherry (2021) by Programming Throwdown Podcast (trivia about Frank, Timely, and Materialize)