Make lmdb optional

[hyanwong]

It would be nice to remove complex dependencies from tsinfer. I think lmdb is one of those, and now only used for legacy support. Could we therefore make this an optional dependency?

[jeromekelleher]

I don't see why not, but there would probably be some semi complex plumbing involved. It's only used in the SampleData, I guess? Or are we using it for storing AncestorData also?

[benjeffery]

We're still using it for AncestorData as it uses the same superclass lmdb code as SampleData. Would be pretty straight forward to switch AncestorData over to a directory store. However we probably want to keep backwards compatibility which would add complexity.

[jeromekelleher]

It would be nice to do, but it's not a priority. LMDB has served us well apart from a few install headaches.

[hyanwong]

However we probably want to keep backwards compatibility which would add complexity.

Do we really want to keep backwards compatibility for AncestorData files? Personally I wouldn't bother. We are still only at 0.4, rather than a 1.0 release.

[jeromekelleher]

I don't think the issue is backwards compatibility, more that we're still using lmbd for the current version of AncestorData. Do I understand this correctly @benjeffery?

[benjeffery]

Yes, that's correct. We'd need to change AncestorData over to a different store.

[jeromekelleher]

I think we have enough to do for now, let's leave it as it is.

[hyanwong]

I'm getting this in the test suite:

tests/test_formats.py: 16 warnings
  /Users/yan/Documents/GitHub/tsinfer/tsinfer/formats.py:106: FutureWarning: The LMDBStore is deprecated and will be removed in a Zarr-Python version 3, see https://github.com/zarr-developers/zarr-python/issues/1274 for more information.
    store = zarr.LMDBStore(

Does this mean we need to remove the internal dependency on LMDB for future use anyway?

[jeromekelleher]

It's not clear when or if we'll move to Zarr v3 so this isn't going to change much. Easy enough to make a store anyway.