lightningnetwork
sqlite db backend
Change Description
Closes #6176
This PR brings SQLite support to LND. As mentioned in #6176, this PR is a first pass implements sqlite support in a somewhat naive way.
It builds on the postgres support work that was added recently, however to avoid any premature abstraction, I've intentionally duplicated some things between the postgres and sqlite kvdb modules. Specifically, the kvdb/sqlite/readwrite_*.go files are nearly completely identical to the ones in the postgres package, and (thanks for golang's database/sql library) could likely be used as-is for future SQL-based kvdb backends.
The table schema used by sqlite is almost identical to the postgres table schema. It is still very much a key-value approach as opposed to a properly normalized SQL table structure. The intention is that moving to a more normalized table structure could come in the future, along with tooling to migrate from the existing tables or from boltdb, but that would likely come in a future PR.
We're also making use of the https://github.com/mattn/go-sqlite3 package (which relies on C bindings) because of its maturity, as opposed to the less mature, but pure golang sqlite driver: https://pkg.go.dev/modernc.org/sqlite. If requiring CGO is a problem, then we could attempt to use the modernc sqlite package instead.
Tasks:
- [x] Get basic SQLite support working via unit tests
- [x] Get integration tests passing with sqlite backend
- [ ] Wire up lncfg support for sqlite
- [ ] Do manual integration testing with the sqlite driver
Steps to Test
TODO
Pull Request Checklist
Testing
- [ ] Your PR passes all CI checks.
- [ ] Tests covering the positive and negative (error paths) are included.
- [ ] Bug fixes contain tests triggering the bug to prevent regressions.
Code Style and Documentation
- [ ] The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
- [ ] Commits follow the Ideal Git Commit Structure.
- [ ] Any new logging statements use an appropriate subsystem and logging level.
- [ ] There is a change description in the release notes, or
[skip ci]in the commit message for small changes.
📝 Please see our Contribution Guidelines for further guidance.
👍
Would it be possible to adapt database replication within this inital PR?
Like: db.location=/location1/sqlite3.db:/location2/sqlite3.db
Would it be possible to adapt database replication within this initial PR?
@blckbx good question! For the first iteration, I was hoping to keep things simple and not bring that sort of complexity into lnd.
There is a new project (which I've only read about, but haven't used yet) called litestream.io that looks promising as a way to replicate sqlite database to another file locally or to cloud storage. That might be a nice solution because litestream runs as a sidecar process and so none of the replication complexity would have to come into lnd itself.
Curious to hear your thoughts!
Would it be possible to adapt database replication within this initial PR?
@blckbx good question! For the first iteration, I was hoping to keep things simple and not bring that sort of complexity into lnd.
Absolutely true. Best to move this to another PR/discussion. But to get back to litestream ...
There is a new project (which I've only read about, but haven't used yet) called litestream.io that looks promising as a way to replicate sqlite database to another file locally or to cloud storage. That might be a nice solution because litestream runs as a sidecar process and so none of the replication complexity would have to come into
lnditself.
Unfortunately I'm not knowledgable enough to recommend a definite solution. So I just point to the issue CLN devs experienced using litestream, namely crashes. In the end they went away from litestream and implemented direct sqlite3 replication instead.
Hey @seejee are you still interested in continuing this PR? (if not we can pick it up)
Replaced by https://github.com/lightningnetwork/lnd/pull/7251
Thanks for getting this started!