Awesome Distributed System Projects

This repository contains list of distributed system projects with open source code in various programming languages which may be useful in order to better understand how to build distributed services.

Databases

• (Golang) Jocko - Kafka/distributed commit log service in Go. [Serf + Raft]
• (Golang) oklog - distributed and coordination-free log management system for big ol' clusters [Archived]
• (Golang) Kingbus - a distributed MySQL binlog store based on raft [Raft]
• (Golang) elasticell - a distributed HA Redis-compatible NoSQL database with strong consistency and reliability
• (Golang) RQLite - lightweight, distributed relational database, which uses SQLite as its storage engine
• (Golang) CockroachDB - distributed fault-tolerant SQL database built on a transactional and strongly-consistent key-value store
• (Erlang) CouchDB - highly available, partition tolerant, eventually consistent document database . Supports master-master setups with automatic conflict detection.
• (C++) MongoDB - document database designed for ease of development and scaling
• (Java) Apache Cassandra - a highly-scalable partitioned row store. Rows are organized into tables with a required primary key
• (Java) Apache HBase - Hadoop database, a distributed, scalable, big data store. Useful when random, realtime read/write access to big data needed
• (Rust) TiKV - distributed transactional key-value database, originally created to complement TiDB
• (Golang) Tair - high-performance and high-availability distributed fast-access memory (MDB)/persistent (LDB) storage service
• (C) memcached - high performance multithreaded event-based key/value cache store intended to be used in a distributed system

Locking

• (Golang) etcd - distributed reliable key-value store for the most critical data of a distributed system [Raft + gRPC]
• (Java) Apache Zookeeper - highly reliable distributed coordination
• (Golang) chubby - A (very simplified) implementation of Chubby, Google's distributed lock service

Streaming

• (Java) Kafka - distributed, highly scalable, elastic, fault-tolerant, and secure event streaming platform
• (Python) faust - a distributed stream processing library that ports the ideas from Kafka Streams to Python
• (Golang) Liftbridge - lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system

Schedulers

• (Golang) dkron - distributed, fault tolerant job scheduling system for cloud native environments
• (Python) Celery - distributed task queue
• (Python) Apache Airflow - a platform to programmatically author, schedule, and monitor workflows

Queues

• (Golang) nsq - realtime fault tolerant distributed messaging platform designed to operate at scale, handling billions of messages per day [Raft + gRPC]
• (Golang) Sandglass - distributed, horizontally scalable, persistent, time ordered message queue
• (Golang) dnpipes - distributed version of Unix named pipes comparable to AWS SQS
• (PHP) GatewayWorker - distributed realtime messaging framework based on workerman
• (C++) ZeroMQ - abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more
• (Java) Apache Pulsar - distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API
• (Java) Apache ActiveMQ - high performance Apache 2.0 licensed Message Broker

Search Engines

• (Java) ElasticSearch - distributed, RESTful search and analytics engine
• (Java) Apache Lucene - a high-performance, full featured text search engine library
• (Rust) MeiliSearch - Lightning Fast, Ultra Relevant, and Typo-Tolerant Search Engine
• (JS) FlexSearch - memory-flexible full-text search library
• (Golang) RiotSearch - distributed, Simple and efficient full text search engine
• (C++) Typesense - fast, typo tolerant, fuzzy search engine

File Systems

• (Golang) JuiceFS - Hadoop-compatible AWS S3-compatible high-performance POSIX file system
• (Golang) SeaweedFS - a simple Hadoop-compatible AWS S3-compatible distributed highly scalable distributed file system
• (C) GlusterFS - distributed storage that can scale to several petabytes
• (C++) GlusterFS - highly reliable, scalable and efficient distributed file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.

Service Discovery

• (Golang) sleuth - master-less peer-to-peer autodiscovery and RPC between HTTP services that reside on the same network

Data Processing

• (Scala) Apache Spark - unified analytics engine for large-scale data processing

OS

• (C) HarveyOS - distributed operating system

Frameworks

• (Golang) etcd - framework for distributed systems development. Provides the core requirements for distributed systems development including RPC and Event driven communication
• (Golang) ergo - port of Erlang/OTP approaches in Golang
• (Golang) gosiris - an actor framework for Golang
• (Python) cotyledon - a framework for defining long-running services. It provides handling of Unix signals, spawning of workers, supervision of children processes, daemon reloading, sd-notify, rate limiting for worker spawning, and more.
• (Java) atomix - fully featured framework for building fault-tolerant distributed systems [REST + Raft]
• (Kotlin) orbit - virtual actor framework for building distributed systems
• (JS) hemera - A Node.js microservices toolkit for the NATS messaging system [RPC]
• (Python) Tooz - centralizing the most common distributed primitives like group membership protocol, lock service and leader election by providing a coordination API helping developers to build distributed applications
• (C++) Nebula - powerful framework for building highly concurrent, distributed, and resilient message-driven applications

Components

• (Golang) Dragonboat - a high performance multi-group Raft consensus library in pure Go
• (Golang) Golimit - Uber ringpop based distributed and decentralized rate limiter
• (Python) Tenacity - general-purpose retrying library
• (Elixir) ex_hash_ring - pure Elixir consistent hash ring implementation based on the excellent C hash-ring lib
• (Elixir) raft - Raft consensus implementation
• (C++) NuRaft - Raft implementation derived from the cornerstone project

Other Resources

• awesome-scalability - Reading list for illustrating the patterns of scalable, reliable, and performant large-scale systems
• awesome-distributed-systems - curated list on awesome material on distributed systems