core icon indicating copy to clipboard operation
core copied to clipboard

Tracking issue: SQL migration

Open goto-bus-stop opened this issue 11 months ago • 3 comments

üWave is moving away from MongoDB to simplify operations (and probably get a performance boost to boot). For a small self-hosted service operating on relational data, MongoDB was a poor choice.

Initially, the focus will be on SQLite, which will have no trouble handling a typical server with a few dozen simultaneous users despite being synchronous. We could use worker threads to improve concurrency later if needed.

#637 contains a large chunk of the work, migrating all MongoDB uses to SQLite.

Follow up items, roughly in order:

  • [x] Typed JSON
  • [ ] Set up LiteFS on fly.io for the demo server
  • [ ] Redis is randomly used for some booth / waitlist / online users related storage. probably just stuff that into a simple key-value table in the database to make the whole program easier to run.
  • [ ] Use SQL transactions where necessary--with mongodb there are no transactions so a bunch of code YOLO'd consistency, but with a SQL database we can actually guarantee transactions easily.
  • [ ] put the migration code into a standalone migration script so I can release the next version without depending on mongo (you would probably run npx u-wave/core#migrate to move over your database, or something?)
  • [x] performance test it with https://wlk.yt/ data. I expect this will be dramatically faster...
  • [ ] Support for other SQL databases, probably Postgres and/or MariaDB, which have relatively simple deployment options and which many self-hosters have experience with

goto-bus-stop avatar Nov 22 '24 21:11 goto-bus-stop