heed icon indicating copy to clipboard operation
heed copied to clipboard

Checking for already opened LMDB Environments

Open darnuria opened this issue 2 years ago • 1 comments

Heed strive to pursue opening LMDB safely, so a database environment must be opened only once inside a process address space.

Any subsequent opening shall return the corresponding already opened environment.

It was done with Path Canonization but it fails to tackle symlinks, hardlinks and renaming/move of the whole database environment (both locks and database).

In an ideal world the environment already open checking must prevail:

  • [ ] The solution should avoid keeping the environment "open" aka holding a file descriptor elsewhere see https://github.com/meilisearch/heed/pull/179#issue-1795327221
  • [x] First opening
  • [x] Opening with the same path
  • [ ] Access through symlink
  • ~[ ] Access through hardlink~ Not possible since hard_link only apply to files and we track dir for envs.
  • [ ] Renaming

Related pull requests/ Issue

  • Idea based on https://github.com/meilisearch/heed/issues/145#issuecomment-1384539350
  • #179
  • LMDB assumptions: https://github.com/meilisearch/heed/issues/20#issuecomment-1378863562

Open question

Is Canonization still pertinant if we check already-openned by Unix: (device/inode) Windows: (Driver/fileID) through same_file?

How to do it compatible with windows without lefting file open.

Problems

Lmdb rely on Unixes on posix filelocking through fnctl

darnuria avatar Jul 10 '23 13:07 darnuria

@Kerollmops would suggest removing from 0.20.0 this issue the solution may cause more trouble than the initial issue.

darnuria avatar Jul 20 '23 16:07 darnuria