Persistence storage engine

[matsjj]

Currently everything is just saved in memory using InMemoryDBHandler, which basically just mocks a database.

For actual usage we obviously need a way to persist state across application restarts.

Possibilities:

• Use JDBC directly and wrap everything into PreparadStatements. Easy to implement, but lots of boilerplate code that needs maintenance whenever one of the data structures change
• Use some ORM system that automatically manages the database according to the object structure
• Serialise the InMemoryDBHandler to disk on shutdown and read it from disk again when booting up. Might be the easiest solution to implement and maintain

[jprupp]

Hey Mats,

I’m not extremely happy with persisting InMemoryDBHandler to disk at shutdown unless one can be absolutely sure that there will not be any data loss if this serialisation does not happen, say, if the program crashes or the power cord is yanked. I think using an ORM that will be synced to every important transition in the program state makes more sense.

Cheers!

On 13 May 2016, at 10:54, matsjj [email protected] wrote:

Currently everything is just saved in memory using InMemoryDBHandler, which basically just mocks a database.

For actual usage we obviously need a way to persist state across application restarts.

Possibilities:

Use JDBC directly and wrap everything into PreparadStatements. Easy to implement, but lots of boilerplate code that needs maintenance whenever one of the data structures change Use some ORM system that automatically manages the database according to the object structure Serialise the InMemoryDBHandler to disk on shutdown and read it from disk again when booting up. Might be the easiest solution to implement and maintain — You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/blockchain/thunder/issues/6

[matsjj]

Yes I think so as well. After taking a deep look into various ORMs and opinions, I saw that a lot of people actually still prefer to write manual SQL with JDBC! But also a lot of the hate and arguments you find against ORMs tend to be from <2011, and are also more performance related in applications where you would find very complicated queries that are impossible to map in an ORM.

For us we have a very basic mapping that revolves more about storing state to disk than to perform difficult queries. Hibernate seems to be the de facto standard, the biggest problem now is how to integrate it nicely into the application flow.

I am fine with annotating the relevant data structures (Channel, RevocationHash, PaymentWrapper, ...), but I don't want to mix Hibernate-specific statements into business logic, so need to wrap it into the DBHandler somehow.

[kushti]

Why not some transactional embedded key-value database? It is almost as simple as Java collections. E.g. MvStore (http://h2database.com/html/mvstore.html) or MapDB (http://www.mapdb.org/)

[jprupp]

I like it, the simpler the better.

On 16 May 2016, at 21:03, Alexander Chepurnoy [email protected] wrote:

Why not some transactional embedded key-value database? It is almost as simple as Java collections. E.g. MvStore (http://h2database.com/html/mvstore.html http://h2database.com/html/mvstore.html) or MapDB (http://www.mapdb.org/ http://www.mapdb.org/)

[jprupp]

What do you think about this?

https://github.com/jOOQ/jOOQ https://github.com/jOOQ/jOOQ

On 16 May 2016, at 21:03, Alexander Chepurnoy [email protected] wrote:

Why not some transactional embedded key-value database? It is almost as simple as Java collections. E.g. MvStore (http://h2database.com/html/mvstore.html http://h2database.com/html/mvstore.html) or MapDB (http://www.mapdb.org/ http://www.mapdb.org/)

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/blockchain/thunder/issues/6#issuecomment-219530893

[matsjj]

Currently looking into MvStore.

Have to be careful to handle Serialization correctly though, some of the bitcoinJ classes behaved weird last time I tried it (even though they are all marked as Serializable).

As far as I can see, jOOQ is database driven, meaning we have to define the schema first and then map our objects into it. Might work, but quite some overhead, given that we are still unsure about some details of data structures.