Data persistentance question?

[kethan]

Hi can I persist data to Amazon s3, Google drive, Dropbox etc ? If so how do I do ?

[BobDickinson]

Similar issue. Would like to load my "db" from object store, jam it into nedb, do some operations, serialized it out of nedb, and write back to object store. How do I serialize in and out of a in-memory nedb instance?

[martinsookael]

If you set it up correctly, the data in your NeDB db is persistent anyway, so you would not need to use outside data services for persistency, just for backuping.

Not sure I understand all of your requirements, but I'd solve it like this:

1. Write a cron job using npm "cron" to trigger every n days
2. The cron job would read the data from NeDB. After reading the data is in JSON format.
3. Save the JSON to a cloud service of your preference. Most SDK-s should be able to save JSON @BobDickinson does this answer the serialisation question?).

If something happens, I'd just get the latest backup.

[marcusjwhelan]

@martinsookael isn't nedb not stored as JSON? I thought it was an object per each line, no commas just a return then the next object.

[martinsookael]

AFAIK* it's stored as JSONL: http://jsonlines.org Where object elements are separated by a newline rather then commas. It's supposed to give you faster search times then searching from the regular JSON**.

• & ** - I haven't been able to find the source on how I know this, I was just recently looking for it in another thread :)

[BobDickinson]

In my case I have a "carrier scale" application supporting on the order of 100m users with a massively parallel chunk of middleware (where this db access will happen). The only persistence supported by the architecture is the object store. My brute force plan was to run an in-memory instance, and at the end of my transactions, save the db/table to the object store, then on subsequent access by that user, load that data from the object store back in to a new in-memory instance. I don't really see a clean/easy way to do that (other than just brute force, per record serialization). I have considered using a temp local disk file as another approach (I'm not sure if copying the object store image to a local file, interacting with it, then serializing that back to the object store, is viable in my architecture, and if that's any better that the brute force per-record "serialize it yourself" approach).

[martinsookael]

@BobDickinson Not sure I understand all of this, but do bear in mind the RAM memory implications.

NeDB loads the whole opened DB to RAM, so if your DB is 30 MB, it occupies 30 MB of RAM.

BUT, If your app opens 30 MB of data (for serialising), then copies the 30 MB of data into NeDB, then just for a moment there you might need 30 (open) + 30 (copy) + 30 (NeDB) = at least 90 MB of RAM for this operation.

Sometimes it might take even more memory, check out these: https://github.com/louischatriot/nedb/issues/483 https://github.com/louischatriot/nedb/issues/472

So far I'm getting great results out of NeDB, but only if the DB in use itself is rather small.

I have considered using a temp local disk file.

I think, that's what NeDB is all about - a local disk file with an input and output as JSON. So your win is, that you don't have to write the input and output yourself.

[JamesMGreene]

By default, a persistent NeDB datafile's format is ndjson (newline-delimited JSON). However, the moment you add a custom afterSerialization and beforeDeserialization hook functions, that default format completely goes out the window at the whim of your hook functions.

[BobDickinson]

In my case, I'm not worried about RAM (the data sets aren't huge, and because it's only loaded for the duration of a transaction, even 3x isn't a problem).

I guess what I want is maybe not in the mainline use case. I'd like to populate a new instance from a stream that I provide, twiddle the db for a bit, then be able to serialize it back out when I'm ready (never touching the local disk).

[salmanff]

Hi can I persist data to Amazon s3, Google drive, Dropbox etc ? If so how do I do ?

I wanted to pick up on the original question as I understand it: ie Instead of using "fs" to store data, has any one considered changing the code so that nedb writes to s3, or dropbox etc? This would be useful if an app is deployed to heroku or openshift for example. Such services store node apps in containers and files written using fs are not persistent. So alternate file systems would be needed to deploy nedb on the likes of heroku and openshift.

Is there an elegant solution to this? For example, some form of thirdPartyStorage option would have to be integrated in the main Datastore object along with the fs credentials (eg dropbox token), and then passed onto persistence.js to call on the alternate file system via a translator js file which has the same functionality as storage.js, linked to the alternate file system. This would be quite cumbersome. Alternatively, afterSerialization and beforeDeserialization could be used to just create the side-effect of writing or reading the data from an alternate file source. Not very elegant either.

Has any one tried to build such functionality?

thanks