cheatcode
@joystick.js/db
I think it's time to do this. It makes a lot of sense to build a simple but reasonably-scalable database into Joystick. Something that's natively Node.js/JS and handles the majority of database queries you'd execute.
The goal is to build this with a query language similar to MongoDB so that writing queries feels like writing code. It should also be able to run on fairly inexpensive hardware, so you could scale up a cluster on some cheap DO Droplets.
See the wishlist in your local repo for notes on features.
If everything works fairly well, goal is to make this the default DB for Joystick > MongoDB. Something that a SaaS developer could starting using and growing their app without thinking too much about it. Have a simple CLI that makes it easy to import/export data in formats that can quickly be migrated to another database if scale requires it.
Thinking about this a bit more. I think a better way to approach this is to just have it be an in-memory database that just lives inside of Joystick. So, instead of trying to build yet-another-database, have it be a lightweight DB that loads into memory but can persist to disk (think a simple version of Redis but with the querying power of MongoDB).
Why? Most apps don't generate a massive footprint. Even with millions of documents, you may only ever use a few hundred MB's of data. There's an opportunity here to make it so that you can use Joystick DB as an in-memory store and then "sync" data quietly in the background to disk (and/or offer an on_write hook that lets devs offload data to a third-party database like MongoDB or PostgreSQL).
The goal would be to give devs a lightning-fast database with a nice query language, but without the overhead of wiring together Redis and another database.
The only thing I want to keep in mind is that if necessary, you should be able to run a Joystick app with this DB installed and use that as your database server (i.e., roll it out on a VPS with a good chunk of memory). Just need to think about how you could do this with minimal effort (e.g., via the config.databases array in your settings.env.json file).
I'm really warming up to the idea of having just an in-memory store with a persistence layer and then having really simple/easy APIs for doing backup restore. At the end of the day, that's really what you care about: it's fast and you don't lose data (and if there is a crisis, recovery is fast/simple).
To make this a really clean DX, I want to make it so that Joystick automatically "starts" the database if this is declared in their env settings. When doing so, Joystick should leverage the .joystick/data folder like .joystick/data/joystick_<env> (where joystick is the name of the db and <env> is either development, staging, or production).
Then, in the settings env file, you could set an option like replicate_to and pass an array of strings like ['staging', 'production'] that could identify copy targets for data in the current env:
{
"config": {
"databases": [
{
"provider": "joystick",
"users": true,
"options": {
"replicate_to": ["staging", "production"]
}
}
],
"i18n": {
"defaultLanguage": "en-US"
},
"middleware": {},
"email": {
"from": "",
"smtp": {
"host": "",
"port": 587,
"username": "",
"password": ""
}
}
},
"global": {},
"public": {},
"private": {}
}
This is coming to mind again. I think the best way to think about this idea is that it's a simple, in-memory database with a query language similar to MongoDB. It also can have hooks so that when you write to the cache, you can hook into another database to persist it. So, you write your queries with Joystick DB, but ultimately, still persist data in your DB of choice.
Why this makes sense: it's an incredible performance trick. Because we're effectively just reading/writing from a cache, our basic UX is really fast but with hooks, we don't have to worry about data not being persisted. We could even have an option on_write_preference that can be either 'memory' (default) or 'disk' such that if it's 'disk', we write to the disk first and then the in-memory store.