PumpkinDB
Problem: working with storage data is not zero-copy
This issue is mentioned in ad6da5243d4f5804592d9bd7fa67306e83a53f63
There was a realization that Env<'a>
wants all referenced elements on the stack to allow have
the same lifetime of 'a. However, the lifetime of the
values extracted through lmdb is limited by the lifetime
of the transaction.
Potential solution:
This means that right now we have to resort to copying values. However, I suspect there's still a chance we can have an optimization here. We might be able to have a temporary "transaction context" stack that has a different lifetime. The feasibility and consequences of such an idea are still to be researched. But if defined well, this should definitely improve the overall quiality and performance of PumpkinDB.
Another solution is to transmute lifetimes and do "transaction exit" scans, but that's pretty hacky.
Another solution is to introduce the concept of "linked Envs" where an Env can have a parent Env, effectively separating them by lifetimes.
Here's a thought. Based on my recent work on the SPDK binding, we can kind of forego this whole zero-copy issue. How so?
SPDK allows to communicate with the NVMe controller bypassing the kernel by pointing to a pinned allocation of host memory directly. Therefore, if we design our own direct storage format (which is, of course, a huge undertaking!) and allocate the heap in such a piece of memory (using spdk::Buffer from the binding), then not only we'll have what essentially amounts to zero-copy reads, but also zero-copy writes!
Obviously, this won't solve the zero-copy reads issue for the lmdb backend, but if we assume that you have to make certain sacrifices if you want performance (like buying a good NVMe SSD), then it's a solution! And if ever get around it, we can still work on the above ideas for adding zero-copy reads to lmdb-based engines.