MeshCentral
MeshCentral copied to clipboard
Ylianst
Deprecate NeDB in favour of another embedded db
NeDB is no longer supported and hasn't been supported for some time, but it is the default embedded database (I'm assuming due to its compatibility with mongodb).
Acebase is a current nosql (firebase) compatible embedded database that uses (among other options) sqlite as the backend storage, meaning that (theoretically) the proven performance and reliability of sqlite can be leveraged with mongodb syntax queries.
Another option is using sqlite "as is", as it has full support for JSON now and can be used as a document database.
Of course NeDB can be considered "good enough" and serious use should switch to mongo anyway, but it would be nice not to rely on an unmaintained package for the core database.
I agree with this. The benefit of NeDB is that it was "Pure NodeJS" and so, there was no risk for failing to install on any platform. I was also sort-of compatible with the MongoDB API. This said, a lot of people use MeshCentral with the embedded database and so, getting something more robust would be welcome.
Well, that took my entire Saturday, but I added basic support for AceBase. As expected, it's a local database with no dependencies. There is plenty more work to do to test it and to improve performance, but the basics seem to work. I am not going to make it the default database for a while, but we can start testing it.
This will be in MeshCentral 1.0.66 when it next comes out.
So, an update on this topic. I spent the weekend adding AceBase and at first it looked good, it's a pure NodeJS solution and it's supported. However, after running it for a while on my development server I started noticing issues. It does not handle corrupt data well and removing records from the database is terribly slow. Removing 1000 old records in the events table can lock up the server for over a minute... like REALLY slow. It looks like each time I remove a record, it rebuilds the database indexes. The indexes are also huge. My tiny database with 2000 records (mostly events) has an index file of over 100 megabytes.
So, I am not sure I want to trust people's data to AceBase, even for small servers. I am going to keep the code there in case someone wants to use it, but I am not going to be using AceBase as the next NeDB.
I started working on SQLite support and that seems to be going well so far. It's not pure NodeJS and has a lot of dependencies, but it seems to be fast and reliable. I will likely complete support for it today and see how well that works.
You could look into pouchdb https://pouchdb.com/ It supports local dbs, external couchdb servers and even adapters for other engines like sqlite https://pouchdb.com/adapters.html
One more to add :) I am almost done with SQLite. I will get that done first.
Just published MeshCentral v1.0.66 with AceBase support, but I changed the graphic to "trial" as I am not super happy with it. I did get SQLite working and that works a lot better.
Honestly I suggested acebase because it's new and worked "well enough" for when I tinkered with it, but sqlite would be a much more battle hardened choice anyway. SQLite is the "de facto" embedded database, it's not going out of support any time soon. With the addition of JSON capabilities, it's probably the better choice regardless.
Hey @Ylianst great to see SQLite support being offered...! I experimented with moving to Postgresql but had to fall back to Mongodb due to plugin Scripttask relying on it (or NeDB).
I understand that plugins are not officially supported, but it would be great if we could work out between @ryanblenis, Meshcentral developers and community how plugins can reliably interact with the database without needing db-specific handling (as is the case currently in Scriptask in db.js).
I really just have to bite the bullet and merge Scriptask in MeshCentral. It's used by many and I have to make it something that is officially supported.
Yeah SQLite has been getting a lot of airtime on HN lately, specifically Fly.io's blog items.
https://fly.io/blog/all-in-on-sqlite-litestream/ https://fly.io/blog/sqlite-internals-btree/ https://fly.io/blog/sqlite-internals-rollback-journal/
And some projects working on replicating SQLite benath the DB:
https://litestream.io/ https://github.com/superfly/litefs
Exciting times to begin using SQLite :)
In having enabled AceBase and SQLite within the same week, I can tell just how much better and more reliable SQLite is. The airtime is certainly justified.
@Ylianst can you submit the issues you ran into with AceBase so I can take a look at them? Huge indexes should only be there if the amount of data indexed is huge, there is not much overhead in the index files. Also, deletion of 1000s of records should be fairly quick, so I'm curious how you are using the db. Kindly note that every database engine has the potential to be quick and slow at the same time, based upon how it's used and the optimisations in place. Would really love to help iron out any issues you ran into!
@appy-one @Ylianst is currently away on vacation, https://github.com/Ylianst/MeshCentral/discussions/4460 he will reply as and when he can 👍