Write Forwarding

[benbjohnson]

Currently, LiteFS supports a single primary node that performs all the writes. However, there are situations where it would be useful to have multiple nodes that can write—even if it means taking a performance hit. Two common examples are background jobs & out-of-band migrations.

This could work by having the primary handoff the write lock to another node temporarily:

1. Given N₁ is the primary and N₂ is a replica.
2. N₂ sends a request to acquire a write lock from N₁.
3. N₁ acquires the write lock on behalf of N₂ and holds it for the duration of the request.
4. N₂ ensures it has the latest transaction data.
5. N₂ executes its write transaction locally.
6. N₂ sends the LTX file for the transaction back to N₁
7. If N₁ still holds the lock, it commits the transaction and notifies N₂
8. If N₁ no longer holds the lock or is demoted, it rejects the transaction and notifies N₂

It is to be determined exactly how the lock handoff is requested by the client application. It could be transparent but that could cause users to experience slow performance if they are not correctly forwarding writes when they can. Maybe this should be a flag in the config to enable it?