Releases?

[pyrossh]

Is is possible to get releases for different OS's (linux primarily) as I don't have go installed? I Just want to test it out and don't want to install the entire go toolchain for that.

Anyways the project seems great. Now I need to build an ORM on top of it and test it out. In our projects we generally need to have most of our working set in memory so this seems ideal but I see that bunt db uses locks so won't it be slow to do multiple writes on the same key? and How does it handle concurrency? and wouldn't having disk persistence reduce the number of writes per second (like redis AOF)?

[tidwall]

@pyros2097 Thanks for the suggestion. I just published pre-built binaries.

I see that bunt db uses locks so won't it be slow to do multiple writes on the same key?

The BuntDB write locks are generally very fast. On my Macbook Pro I can consistently get +250,000 write/sec when writing random keys with persistence on. When persistence is turned off it's +500,000. The fastest is with persistence off and writing sequential keys which is +1,000,000.

BuntDB is transactional so each write that I mention above is basically:

MULTI
SET key value
EXEC

But it's possible to have many writes wrapped in a single transaction and this will increase the performance a lot.

How does it handle concurrency?

It's multithreaded. Go is very good with concurrency. The penalty for multiple concurrent reads is miniscule. BuntDB can easily achieve many millions of reads per second. Writes only have the lock to contend with as stated above.

wouldn't having disk persistence reduce the number of writes per second (like redis AOF)?

Yes. The number of writes per second are reduced with persistence turned on.

It may be worth noting that SummitDB is basically Raft wrapped around a BuntDB database. By far the slowest aspect to this configuration is the handshakes between servers to keep the system consistent.

As far as the performance of Raft+NoSQL servers, SummitDB performs incredibly well. But it will likely never be as fast as a single instance of Redis with AOF turned off.

[pyrossh]

Thanks for the great info. I guess this information needs to be put out somewhere. Now I need to dockerize the summitdb using some method like this one https://github.com/pyros2097/docker-images/blob/master/web/prod.sh. Its right now 17mb, I guess with production options I can make it to a 4-5mb Docker Image and also need to include the redis-cli for playing (of course many commands need to be removed maybe a fork should do it) or maybe a UI will be better and easier these days since this has querying capabilities. And then add to my docker-compose stack and test it out with node's ioredis.

The only other thing is? Is it possible to configure the database to only have the working dataset in memory (Like flush out keys from memory which are not frequently used but again that will need some other sort of persistence layer like BoltDB I guess). And why dindn't you use Rust for this shouldn't it be more performant? I fear go's garbage collector when it comes to heavy loads.

Thanks for the DB. Redis was my favorite protocol and database but one thing that dropped me of it for serious use as a Database over postgres was its consistency and no rollback feature and things like these, https://muut.com/blog/news/april-2014-service-failure.html https://muut.com/blog/technology/redis-as-primary-datastore-wtf.html

I even planned on implementing redis over cockroach but to no avail. https://github.com/mjibson/cockroach/tree/redis https://github.com/cockroachdb/cockroach/issues/587

[tidwall]

Now I need to dockerize the summitdb using some method like this one

My other project Tile38 is in the process of being dockerized. We are using the alpine base image and it seems to generate a pretty small binary over all. Perhaps that could be a starting point for SummitDB? I'm not well versed in Docker so I'm up for suggestions.

Is it possible to configure the database to only have the working dataset in memory (Like flush out keys from memory which are not frequently used but again that will need some other sort of persistence layer like BoltDB I guess).

Right now the working dataset operates entirely in memory. The persistence is required to keep consistency and durability up to the Raft standards.

Unfortunately SummitDB (and BuntDB) does not have an automatic flushing mechanism for removing infrequently used keys. This sounds a lot like LRU cache evictions similar to what is possible in Redis. I suppose this could be a possible feature in SummitDB in the future. The biggest challenge that comes to mind is operating the evictions across the cluster.

And why dindn't you use Rust for this shouldn't it be more performant? I fear go's garbage collector when it comes to heavy loads.

Good question. A day does not go by that I do not ask myself the question "Why not Rust?".

Yet, it's my opinion that if I had chosen Rust for this project then this conversation would happen sometime in the future, and the question might be "Why not Go?"

The performance gains of Rust over Go is rather small. I feel the same way about C over Go for that matter. The Go GC stops are minor in my opinion. I've never noticed a slow down that has affected performance, or have I heard of any complaints from users in production for any of my projects. It just hasn't been an issue for my projects yet.

Redis was my favorite protocol and database but one thing that dropped me of it for serious use as a Database over postgres was its consistency and no rollback feature and things like these, https://muut.com/blog/news/april-2014-service-failure.html, https://muut.com/blog/technology/redis-as-primary-datastore-wtf.html

Redis is my favorite too and I created SummitDB for the exact reasons you mentioned. In addition I wanted built-in indexing and r-tree support.

Thanks for the excellent feedback.