Anyone actively using Hyperdex?

[davedbase]

Hi,

I was wondering if anyone is actively using Hyperdex? Love the project and everything about it, very sad that it's no longer maintained :(

Dave

[rescrv]

Hi Dave,

HyperDex is no longer maintained, but that's because circumstances have made me switch my attention to another project: Consus.

Consus is built using the same dependencies as HyperDex (which is why there is still some activity on them), and built using many lessons I learned working with people who were trying to use (and succeeding or failing to varying degrees) HyperDex.

Consus has two (three) differences from HyperDex that I believe will make it more appealing in the long run:

1. Consus doesn't separate the implementations of transactions and the regular code. Once we have a paper public on the topic and I have clearance to release the code, all of the code will be available under the same license, with no commercial fork.
2. Consus provides wide-area, geo-replicated transactions with much lower latency than is possible in any other system I know with the same consistency that Consus provides.
3. I'm still actively working on Consus; I'm not actively working on HyperDex.

Of course, this doesn't help right now, but it does give you something to possibly follow in the future.

-Robert

[unquietwiki]

@rescrv I came across this project via Wikipedia. Is http://www.garret.ru/consus.html the Consus you mention, or a different Consus?

[aXe1]

@unquietwiki According to @rescrv 's twitter: consus.io.

[rescrv]

It is indeed http://consus.io

[unquietwiki]

@rescrv Thanks for that: I updated the wiki page with the new info. Question going forward: should folks use this project at all, or consider taking up maintenance of it? At the very least, a Debian/Jessie & Ubuntu/Xenial package for the server might be useful for existing projects. Also curious if this has any overlap with https://rethinkdb.com/ , which is another recently orphaned document store.

[rescrv]

I've stopped developing the project for non-public reasons that aren't technically related to the project.

I'd be happy to hand off or publish the build scripts etc that I used to generate packages so that someone else can keep the project building, re-roll 1.8 with fixes I've applied since then, and package with the up-to-date dependencies that fix many bugs.

[unquietwiki]

@rescrv Publishing the build scripts + fixes would be helpful. I just found this project myself, looking up usable data stores for a project: I might be able to help out more if I can get it working on my end.

@ddibiase @aXe1 What about either of you? Or know anyone else helping in maintaining this? We could also try finding someone via a proper subreddit. https://www.reddit.com/r/programming/comments/a6qgz/proggit_im_trying_to_compile_all_the_known/ has a list of possible candidates.

[davedbase]

@unquietwiki you have no idea how badly I'd like to help out, but I've got a stack on my plate right now. Also dbs aren't my forte but I'd be willing to learn/support other peoples efforts. Would love nothing more than to see a stable/maintained HyperDex codebase.

I'm wondering if maybe the author of LevelGraph might be interested. There's a LevelHyper adapter that I think he was involved in porting.

[unquietwiki]

@mcollina , @ddibiase and I were trying to help @rescrv figure out how to maintain this, since he's working on another related project now. I have some time and ability to help out, but I just discovered this project yesterday. Unless I understand incorrectly, it seems there is some convergence via https://github.com/rescrv/HyperLevelDB ?

[rescrv]

I'm actively maintaining the dependencies below HyperDex, so those dependencies will be easy to keep going. I've started pulling my build scripts for HyperDex from my backups. It may take past the holiday before I can make progress on this issue.

[rescrv]

I've pushed the HyperDex build process to GitHub. You'll need to clone the rescrv/minion repo and have Docker and go installed. In the minion repo, run "make" to build the "minion" and "minion-daemon" binaries. Add them to your path.

Next, setup the releng environment. I've uploaded a pre-bootstrapped build environment to S3 at s3://rescrv/hyperdex-releng.tar.gz. It is available to authenticated AWS users. Download that complete environment and untar it. The hyperdex-releng/build directory is a clone of the rescrv/hyperdex-releng repo.

Deploy minion by running minion-daemon in the hyperdex-releng directory. Pass this directory as the -d argument to any minion command. Run the commands in README in that repo to bootstrap your environment and build the tarballs and packages. I made sure it builds all the way through the tarballs and packages. I've disabled some broken pieces I didn't have time to fix.

You can copy the results of the build process with minion -d /path/to/hyperdex-releng get-artifacts dev latest output

Sorry for throwing everything at you quite quickly. I'm happy to answer questions and provide clarification. I just wanted to get this out to the interested folks as soon as possible.

[unquietwiki]

@rescrv Thank you so much for doing this. I plan on trying to make use of this soon, as time allows.

[rescrv]

An update to this issue. I have made Consus (http://hack.systems/2016/12/13/hello-consus/) open source and would strongly encourage anyone considering helping out with HyperDex to get involved in Consus instead. I can provide feedback and guidance for both, but am only going to be actively developing Consus going forward.

[webmaven]

@rescrv, is this the public paper?: https://arxiv.org/abs/1612.03457

[rescrv]

Yes. It and the code are linked from the blog post I pointed to.

[unquietwiki]

@rescrv Thank you for this. You should consider hitting up /r/programming on Reddit with a post about this too.

[rescrv]

You'll notice there is a doc/ directory. Part of rescrv/consus-releng builds this documentation. I need to write the HTML templates to generate the documentation (it currently generates jinja2 templates) and then post it.

After that I'll do wider promotion on social media. I had reasons to open source the code when I did, but I'm not ready to go on a promotional circuit of social media yet (it hit hacker news already).

[higaxman]

@rescrv Here at Forcive (www.forcive.com) we somewhat got interested at HyperDex and we are looking to it as a possible component part for our real time analytic platform. Understanding Consus is the continuator of HpyerDex we are even interesting to contribute to this project.

[rescrv]

@higaxman It's great to hear of your interest. If you're interested in HyperDex, I'd be happy to provide a little assistance getting you off the ground with the build scripts. If you're interested in Consus, I'd point you to its repo.

If you (or others reading this post) are deciding between the two, I'd suggest reading this: Why Consus? What happened to HyperDex?