lnd icon indicating copy to clipboard operation
lnd copied to clipboard

channeldb+invoices: create native SQL schema for invoice storage

Open Roasbeef opened this issue 3 years ago • 1 comments

We recently added postgres support, but in a way that just maps things to a naive key-value table. In order to take advantage of increased queryability, and scalability gains, we should convert the invoices (which can be heavily loaded on large nodes that help to operate services) to a native SQL schema. Along the way, we'll want to abstract away the invoice registry's storage behind an interface, so this new data format can be swapped in at runtime, or during initial configuration of the daemon.

Steps to Completion

  • [ ] #6842
  • [ ] #6954
  • [ ] Create a new SQL-native invoice schema.
    • Tools like sqlc look interesting here as it lets you mainly write SQL in a single file, and have the CRUD code be generated. We use this in some backend products at Lightning Labs now.
    • Alternatively, since sqlc doesn't support sqlite fully (though I think if you don't use any sqlite specific features it works), we could look into using other ORMs.
  • [ ] Implement a migration tool to be run automatically given a relevant set of flags, or manually via a migrate-then-shutdown flag/script.

Related to:

  • https://github.com/lightningnetwork/lnd/issues/6176

Roasbeef avatar Feb 22 '22 01:02 Roasbeef

Started to look into the first two points as of assignment to #6954.

bitromortac avatar Oct 06 '22 10:10 bitromortac

Re-opening since one part is still missing (will link when PR is up).

guggero avatar Jan 17 '23 18:01 guggero