albe
Take lessons from other popular systems
Some notable projects from where to take inspirations or lessons:
- (Get)EventStore: THE stream database for event sourcing - https://eventstore.org/docs/reference/internal-architectural-overview/index.html
- apache kafka: a distributed streaming platform -https://kafka.apache.org/documentation/#implementation
- Microsoft FASTERLOG: a WAL implementation used in FASTER key-value store - https://github.com/microsoft/FASTER/tree/fasterlog/cs/src/core
- rocksdb WAL: https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log
- https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
- Chronicle Queue: https://github.com/OpenHFT/Chronicle-Queue#writing-to-a-queue
A lot of the architecture is already very similar to some of those systems. The API was inspired by EventStore, the storage semantics are very similar to Kafka (coincidentally).
Kafka uses the file offset as the document index id, which saves storing that additionally, but comes at the cost of not being (easily) able to do optimistic concurrency checks, which makes Kafka unfit for an event store, which requires consistency guarantees.
Kafka uses the storage format without transformation as the communication format, which allows sendfile usage and avoids copying of data, when sending it from a node. This is currently not necessary, since event-storage currently has no replication and is single-machine.
EventStore chunks the log file for "operational reasons such as incremental back ups", which makes sense. Maybe the partitions should themself be chunked into multiple files, in which case it would make more sense to keep each partition in an own folder to reduce inode usage.
https://github.com/AxonIQ/axon-server-se/tree/master/axonserver/src/main/java/io/axoniq/axonserver/localstorage