Multiple connections and working with database object

[pzuraq]

Hey,

Loving the driver so far but we are having some issues figuring out coding patterns. We need to access the database from multiple contexts (Say, to load the User in our middleware, then to manipulate some nodes in one of our routes) and this is problematic because we can't connect to the database multiple times. If one db.connect occurs, then all subsequent one's return an empty database object.

Given this, we are wondering how we are supposed to access the database from every context we need. We were trying to export the db object like so:

neo4j.connect(function(err, database) {
    module.exports = database;
});

But this doesn't work because the export is in the callback from an async function.

The only other option we are seeing is to wrap the whole app in the callback from the connect query, literally forcing there to be an active connection in the app. Is this how you do it?

[MatAtBread]

That's how I do it. One connection for the entire app. Note also that you should really only have one connection per system (e.g. cluster) - you can't open the same embedded DB multiple times as the first one locks it for exclusive use. I believe it's possible to open it for subsequent read access, but haven't tried it.

[pzuraq]

@MatAtBread If that's the case how would you use the database for concurrent reads and writes? We expect our app to be accessed by many users at once and we were planning on creating a pool of connections so that we wouldn't have conflicts when using transactions and such.

[MatAtBread]

You can use the one "connection" to do concurrent reads and writes - it's transaction safe from a JS context (remember JS/node is single-threaded). If you want to cluster your front-ends or middleware, you should use the REST interface, or embedded the Java API with a web API of your own construction (which is what we do).

[pzuraq]

JS is single threaded, but it's also asynchronous so there is no way to know what order functions will be executed in. Is it possible to have transaction overlap because of that? Or does the fact that the library is mostly synchronous mean that won't happen?

[MatAtBread]

You need to lock and unlock at the right times for your application. If you have a sequence of async operations you can lock before they start as long as you ensure you unlock in the final callback (under error and success conditions). Specifically the cypher query routines are asynchronous, so unlock in the callback using a closure to store the transaction in. I don't know what you mean by not knowing what order async functions are executed in - it's true they may not execute in any specific order, but knowing that will execute is enough.

I did try using the request & response to lazily lock and unlock the DB (e.g. store the transaction in the request object, and listen for the response end and unlock there), which works, but performance wasn't great as the locks last for much longer than is actually necessary. As a result, I lock almost atomically to hold locks for the least amount of time possible.

[joewhite86]

Hi @pzuraq,

You are right, concurrency and transactions doesn't work as expected atm. I'm working on the issue with the 2.0 release of the driver. I already have some ideas how to sort that out, but sadly also have too many other things to do atm.

I'll see if I can come up with sth. till next week.

[pzuraq]

@joewhite86 That would be excellent. Currently we are using the REST interface, but we would like to move over to embedded for performance gains at some point. When we do that we will try to contribute.