[feature request] go-memdb backend

[tao12345666333]

I want to know if we can add a new backend using go-memdb as in-memory database.

https://github.com/hashicorp/go-memdb

Provides the memdb package that implements a simple in-memory database built on immutable radix trees. The database provides Atomicity, Consistency and Isolation from ACID. Being that it is in-memory, it does not provide durability. The database is instantiated with a schema that specifies the tables and indices that exist and allows transactions to be executed.

The database provides the following:

• Multi-Version Concurrency Control (MVCC) - By leveraging immutable radix trees the database is able to support any number of concurrent readers without locking, and allows a writer to make progress.

• Transaction Support - The database allows for rich transactions, in which multiple objects are inserted, updated or deleted. The transactions can span multiple tables, and are applied atomically. The database provides atomicity and isolation in ACID terminology, such that until commit the updates are not visible.

• Rich Indexing - Tables can support any number of indexes, which can be simple like a single field index, or more advanced compound field indexes. Certain types like UUID can be efficiently compressed from strings into byte indexes for reduced storage requirements.

• Watches - Callers can populate a watch set as part of a query, which can be used to detect when a modification has been made to the database which affects the query results. This lets callers easily watch for changes in the database in a very general way.

[brandond]

That might be hard, since kine expects a SQL compatible storage engine. Seems like if you want an ephemeral dataatore you could get the same result out of sqlite with the special :memory: database file path?

[tao12345666333]

yep, at the same time I also hope not to introduce other components to accomplish this.

[tao12345666333]

maybe we can using https://github.com/araddon/qlbridge to translate it. https://github.com/araddon/qlbridge/tree/master/datasource/memdb

[tokers]

Introduce an SQLite with the in-memory file path is still a bit heavy for us, we need some further discussions.

[brandond]

What specifically are you trying to achieve?

[brandond]

fwiw, I tried using sqlite in memory mode and it's pretty flakey, due to changes in table locking semantics.

docker run --rm -it --privileged --name k3s-server-1 rancher/k3s server --datastore-endpoint 'sqlite://file:memdb1?mode=memory&cache=shared'

[akutz]

fwiw, I tried using sqlite in memory mode and it's pretty flakey, due to changes in table locking semantics.

Is this still the case? I've recently started exploring https://github.com/kcp-dev/kcp/issues/54#issuecomment-1074142302 how to use kine with kcp as a means to support a light-weight binary CLI program that uses controllers. I was hoping to leverage an ephemeral, in-memory DB like sqlite in-mem for this purpose.

[brandond]

You could just use kine but store the SQLite database in a temporary directory that's not persistent?

[akutz]

You could just use kine but store the SQLite database in a temporary directory that's not persistent?

True, but part of the goal is a super portable, self-contained means of implementing controllers in CLI programs. I'll take your suggestion under advisement. Based on the response, can I assume that in-memory sqlite still does not work well?

[brandond]

You're welcome to try it and see if it works for you. It didn't work very well for a whole Kubernetes cluster using the example command I gave.

[akutz]

Ack. Thank you. So just to be clear, to your knowledge no one has a) tried it since you did last July or b) worked on it since then either?

[brandond]

yep.

Some of the requirements to use the sqlite in-memory store are described here: https://www.sqlite.org/inmemorydb.html - I don't remember specifically what I ran into with locking, but it should be easy enough to reproduce.

[brandond]

Seems to be the same issue described here: https://github.com/mattn/go-sqlite3/issues/50#issuecomment-87811607

[mhmxs]

That might be hard, since kine expects a SQL compatible storage engine.

@brandond Is this statement still valid?

The reason I think it may not, is: here https://github.com/k3s-io/kine/blob/master/pkg/endpoint/endpoint.go#L59 kine creates a raw etcd listener without any sql specific wrapper. It looks simple task to create any backend, not just SQL. As i see a backend only has to implement this interface: https://github.com/k3s-io/kine/blob/master/pkg/server/types.go#L20 and has to start it's own listener.

:pray:

[brandond]

We now have a non-sql backend in https://github.com/k3s-io/kine/tree/master/pkg/drivers/nats, but it was fairly non-trivial for the Synadia folks to implemement. You might look at that as an example though.