Question about environment sharing

[jonsecchis]

trafficstars

Hi,

I'm migrating from mozilla's rkv, since I'm interested in using mdbx and heed also provides all the functionality I need out of the box.

I have a doubt regarding the instantiation of Environments. In rkv there is a Manager interface for operating the database, in order to guarantee that the same environment is not opened twice.

While reviewing this crate, I noticed that you already wrap the environment with an RwLock. So, is it correct to assume that heed will uphold the "single environment per process" guarantee without a need for an ad-hoc manager in my application code? Can I safely open and use heed::Env directly across many async handlers in my application?

I'm fairly new to Rust, so I'm sorry if this is an obvious question.

Thank you!

[Kerollmops]

Hey @jonsecchis,

I'm migrating from Mozilla's rkv, since I'm interested in using mdbx and heed also provides all the functionality I need out of the box.

I would be glad to know which functionalities.

While reviewing this crate, I noticed that you already wrap the environment with an RwLock. So, is it correct to assume that heed will uphold the "single environment per process" guarantee without a need for an ad-hoc manager in my application code? Can I safely open and use heed::Env directly across many async handlers in my application?

You can indeed safely open and use an environment without having to worry if it as already been opened by the same process, I keep a map that binds the canonicalized path of the env with the opened envs, this way we return the already opened env if we tried to open it twice or more.

I'm fairly new to Rust, so I'm sorry if this is an obvious question.

No worries, try to read the source code, then ask anyone if you need more details on the implementation. 😃

[jonsecchis]

Thank you for such a prompt answer, @Kerollmops!

I would be glad to know which functionalities.

My reasons to migrate from rkv and also the reasons why I see heed as the absolute go-to wrapper for (L)MDB right now:

1. Reverse iterators without dealing directly with cursors;
2. Transparent environment management, which makes for cleaner call-site code;
3. Typed interface for Databases that is really ergonomic and flexible;
4. Excellent code quality and clarity;
5. Well maintained;

I see Heed as a gem, a hidden treasure. It is a project that delivers immense value out of the box for the industry. The fact that you use it as the core to Meilisearch is also fantastic news, because it gives us users the peace of mind to rely on it for production systems.

Also, there is much room for the project to grow. With more effort on documentation, for example, I can really see this project blowing up in popularity. LMDB is a hard beast to work with in a safe manner, and Heed simply elevates it to a level of simplicity and usability that is unparalleled right now.

Without Heed I would probably need to migrate to sqlite which is far too limiting for advanced use cases. I'm really thankful for your work. It really moves me.

[Kerollmops]

@jonsecchis, this is so pleasant to receive this kind of answer. You really don't know the time I spend on making heed a better library every day. It will keep being the defacto key-value store MeiliSearch will use in the future, and work will always be done on this library to elevate it again. ❤️

By the way, I'm relying on it to build an information system for an investment bank startup in Brazil.

As a little piece of advice, don't use the mdbx backend for now. It would be best to rely on LMDB and maybe wait for me to bump the libmdbx backend to 0.9.2.

Have a nice day!

[jonsecchis]

@Kerollmops Believe me, when I read the code, its evident that much sweat and dedication was put into it.

As a little piece of advice, don't use the mdbx backend for now. It would be best to rely on LMDB and maybe wait for me to bump the libmdbx backend to 0.9.2.

Yes, I came to the same conclusion after trying the mdbx feature. It didn't even compile, got a couple of: the name [x] is defined multiple times and a cannot find function abort_txn in this scope error.

I don't know why, though. Tried cleaning up my cargo directories and building it again from a clean state but even so, I could not make it pass those compiler errors. I'm settled with the LMDB backend.

I saw in a previous issue that you wanted to break up the repo in two crates, one for each backend. +1 for that.

Also, I would consider a name tweak. Heed never showed up in any of my Google searches. It is considerably hard to find. Although the repository description mentions LMDB, it seems to have zero relevance on searches.

The way I found it was through https://github.com/erthink/libmdbx

[Kerollmops]

Yes, I came to the same conclusion after trying the mdbx feature. It didn't even compile, got a couple of: the name [x] is defined multiple times and a cannot find function abort_txn in this scope error.

I don't know why, though. I tried cleaning up my cargo directories and building it again from a clean state, but even so, I could not make it past those compiler errors. I'm settled with the LMDB backend.

Struggling with the build system of libmdbx was one of the main reasons I want to remove this key-value store's support for heed. But I know that there is a bunch of people who depends on that. Therefore I keep it and will wait for the 0.9.2 of libmdbx version to come out and hopefully fix those build issues.

I saw in a previous issue that you wanted to break up the repo in two crates, one for each backend. +1 for that.

Yes, that is something that I want to do for such a long time but don't quite know where to start, I opened an issue and found out that the im and im-rc crates were doing that by using cargo make but didn't dive much more into that.

Also, I would consider a name tweak. Heed never showed up in any of my Google searches. It is considerably hard to find. Although the repository description mentions LMDB, it seems to have zero relevance on searches.

The way I found it was through https://github.com/erthink/libmdbx

I am not sure how to name it. If you have any name proposition, that would be cool. I tried to find an appropriate name when it was named zerocopy-lmdb back then.

[Kerollmops]

Now that libmdbx is archived and never was correctly supported by heed, I can close this issue and hope the recent pre-release v0.20.0-alpha.1 will please you.