LMDB slasher pre-allocates max DB size on Windows

[michaelsproul]

Description

All current releases of LMDB (up to 0.9.29) require the full size of the database to be allocated on Windows. This means that the slasher will create a 256GB file immediately upon starting!

Details

It turns out the LMDB codebase has two main branches:

• mdb.RE/0.9: stable code deemed worthy of release as part of the v0.9.x series
• mdb.master: experimental/under-development code

The bug we are interested in patching is ITS#8324, which causes LMDB to pre-allocate the entirety of the mapped file on Windows. The bugfix is unfortunately only available on mdb.master at the moment, so no stable release of LMDB exists for us to target. I confirmed that using mdb.master resolves the issue, and the tests pass. For reference, this is the patch for Cargo.toml:

[patch]
[patch.crates-io]
lmdb-sys = { git = "https://github.com/michaelsproul/lmdb-rs", branch = "windows-fix-track-master" }

We can't use this solution in prod because we don't know what gnarly bugs may be lurking in the weeds of LMDB's dev branch. Instead we must migrate to a suitable LMDB release once one becomes available.

The release we want should include NtCreateSection: https://github.com/LMDB/lmdb/blob/4b6154340c27d03592b8824646a3bc4eb7ab61f5/libraries/liblmdb/mdb.c#L4488

[michaelsproul]

Blocked on a suitable release of LMDB

[michaelsproul]

Fixed by switching to MDBX: https://github.com/sigp/lighthouse/pull/2776

[michaelsproul]

Reopening this now that we are planning to support LMDB again.