boosting nedb performance in big databases

[ThisIsMrAli]

Hello @louischatriot I'm using nedb on quite a heavy load and feel like it doesn't run as fast as it supposes to. with every operation (CRUD), nedb tries to read from the database or write to database file. i think its not necessary, we can do some refines in nedb structure to be super powerful in heavy databases. so we should:

1. always have synced database in Ram
2. when we manipulate data(create, update, delete) nedb should update Ram database (not file database).
3. nedb should provide a save function that saves Ram database to file database(even nedb calls saves function in intervals, for example, every 5000ms)
4. nedb should trie to save Ram database to File Database only if Ram database was dirty.

I think if we add these functionalities we will have super performances in huge and small databases! no more fs I/O in every function call. i know I/O is cheap with nedb structure, but it takes time for heavy database, for example for a database with 50K records it takes 1200ms to run a single insert function.

even it's possible to sync Ram Database with File Database during CompactDataFile function.

thanks! ali

[fjeddy]

I disagree. NedB is a drop-in replacement for MongoDB, which does and behaves exactly the way you want to convert NedB.

NedB is meant to be simple, not only in functionality, but behavior, and it's simplicity is the sole reason I'm using it for as much as I do. Making NedB memory-based will instantly require and consume more of the server, you make NedB faster, but at the cost of needing more resources (RAM).

I would suggest that you simply move to, and use MongoDB, as the readme explains that you def. should do when you have such a heavy database as you do.

[devqazi]

@fjeddy Correct me if I am wrong, you said "Making NedB memory-based will instantly require and consume more of the server". Doesn't NeDB already keep whole database in the memory?