Considering Btree as an option for new SQLite backend muxer

[danshearer]

Hello,

Just to let you know that LumoSQL is collecting the very few K-V stores that are broadly compatible into backend storage options for SQLite. We haven't studied this project in detail yet but so far it looks like it is a candidate as much as the others. And as discussed on the SQLite forum some of your design features seem well worth adding to SQLite.

Dan

[malbrain]

Hi Dan,

I haven't done much with the btree code except to timing test a "nano" locking btree (locking skip list pointers) in a skip list page. It was not enough to beat the librarian page discipline layout.

My current btree improvement project eliminates root page read locks by keeping multiple read-only root pages each containing a previous version of the root page.

Could you outline what you mean by backend muxer? or is it intended as a descriptive to multiple interleaved transactions?

[malbrain]

My most current btree code is in the database project which is included in the javascript interpreter project. I'm implementing SSN concurrency.

[malbrain]

finish the initiatives for database and source code for the past year.

1. move key payload from tail of key to new field in keyslot
2. add an append page table index to facilitate promoting keys into fences for new pages
3. integrate MVCC concurrency visibility check for database keys
4. add read only root copies that don't need page latches

[danshearer]

Could you outline what you mean by backend muxer? or is it intended as a descriptive to multiple interleaved transactions?

This has evolved quite a bit. But as things stand now, there are makefile targets that will build SQLite of various versions with each of: LMDB of all versions, BerkeleyDB of one version, or of course the original SQLite Btree. The same makefile also does performance benchmarking and saves the results into a standardised database capturing information about the build and runtime environment, with a simple query tool to compare results.

Big picture then what we have is:

1. A way of trying out new K-V stores with SQLite without going to the effort of creating a potentially dead-end fork. It isn't perfect yet but its a really good start.
2. A way of measuring the effect of these backends, because the matrix of possible versions is extremely large. Any of the combinations can have various SQLite or backend optimisations switched on, for example.

There's plenty of work to be done. Ideally we would have an API interface for K-V stores, but in the absence of that we have a way to make life relatively easy. Once a port has been done once, moving to a different versions of either SQLite or the K-V store is as painless as we can make it at the moment.

[danshearer]

finish the initiatives for database and source code for the past year.

1. move key payload from tail of key to new field in keyslot

2. add an append page table index to facilitate promoting keys into fences for new pages

3. integrate MVCC concurrency visibility check for database keys

4. add read only root copies that don't need page latches

I wasn't able to see this work in the C code under the Btree-source-code git tree?

Thanks

[malbrain]

code so far is under database repository alpha subdirectory. as i work through code changes i'll update the alpha sub directory version in the database project

[danshearer]

Ok I see it: https://github.com/malbrain/database/tree/master/alpha .

Dan