zict icon indicating copy to clipboard operation
zict copied to clipboard

lmdb: Platform-specific default map size

Open hack3ric opened this issue 2 years ago • 0 comments

The current code works fine on x86 platforms since x86-64 uses at least 48-bit of virtual address space. On other 64-bit platforms like aarch64 or riscv64, the minimum allowed virtual address is 39-bit 1 2. Current 2**40 allocation will fail:

zict/tests/test_lmdb.py:53:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <zict.lmdb.LMDB object at 0x40043bfc10>
directory = '/tmp/test_lmdb-sfepfnw8'

    def __init__(self, directory: str):
        import lmdb

        # map_size is the maximum database size but shouldn't fill up the
        # virtual address space
        map_size = 1 << 40 if sys.maxsize >= 2**32 else 1 << 28
        # writemap requires sparse file support otherwise the whole
        # `map_size` may be reserved up front on disk
        writemap = sys.platform.startswith("linux")
>       self.db = lmdb.open(
            directory,
            subdir=True,
            map_size=map_size,
            sync=False,
            writemap=writemap,
        )
E       lmdb.Error: /tmp/test_lmdb-sfepfnw8: Operation not supported

zict/lmdb.py:43: Error

Switching to 2**37 on aarch64 and riscv64 should fix the issue.

hack3ric avatar Oct 31 '23 10:10 hack3ric