Transaction rearchitecture

[thorfour]

This issue is meant to describe how transactions work in FrostDB today, and how we would like them to work in the future. It is meant to act as a way to discuss implementation details for how we might re-architect the code to take what we have today and move it towards what we ideally want for transactions.

Today

Transactions are just a logical incrementing number that indicates a rough ordering of when writes came into the system. Today we do not offer a way to have a multi-write transaction; as such there is no possibility for write tearing. What that means is to the end user transactions are effectively meaningless. Our transactions also don't guarantee read snapshot isolation. Meaning that if you begin a read it is possible to have writes that technically newer than when you logically started your read from being returned in that read (but remember that they will be consistent writes returned not partial writes).

This non-isolation can happen in the following way:

1. Read starts tx = 5
2. Write starts and completes at tx = 6
3. Previous write triggers a compaction
4. Compaction moves the tx=6 into L1 (compaction destroys transaction information)
5. Read from 1. continues and will read the data from the tx=6 write from L1 since it can't know what tx it's from

Today we also use our tx system to order administration events in the system via the write ahead log (WAL). Some of these events include block rotations to persistent storage, and snapshots taking place. Compactions currently are not recorded in the WAL.

A large amount of the above were not active design decisions, but were things that have come out of the tumultuous nature of the active development of FrostDB. Many of these decisions made sense at the time but as the code has evolved they may no longer serve the original purpose.

Hopes and Dreams

What we'd like to see out of our transaction system is:

• Transactions offer snapshot isolation (if you're reading at tx=5 your read only returns writes that are <= tx5)
• Transactions are only used for data change events (only writes into the database create a transaction)
• The WAL and transactions have a 1:1 relationship (the WAL doesn't contain admin events)
• Transactions could be extended to support multi-write transactions (see: https://github.com/polarsignals/frostdb/pull/733 )