zict
zict copied to clipboard
lmdb: Platform-specific default map size
The current code works fine on x86 platforms since x86-64 uses at least 48-bit of virtual address space. On other 64-bit platforms like aarch64 or riscv64, the minimum allowed virtual address is 39-bit 1 2. Current 2**40 allocation will fail:
zict/tests/test_lmdb.py:53:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <zict.lmdb.LMDB object at 0x40043bfc10>
directory = '/tmp/test_lmdb-sfepfnw8'
def __init__(self, directory: str):
import lmdb
# map_size is the maximum database size but shouldn't fill up the
# virtual address space
map_size = 1 << 40 if sys.maxsize >= 2**32 else 1 << 28
# writemap requires sparse file support otherwise the whole
# `map_size` may be reserved up front on disk
writemap = sys.platform.startswith("linux")
> self.db = lmdb.open(
directory,
subdir=True,
map_size=map_size,
sync=False,
writemap=writemap,
)
E lmdb.Error: /tmp/test_lmdb-sfepfnw8: Operation not supported
zict/lmdb.py:43: Error
Switching to 2**37 on aarch64 and riscv64 should fix the issue.