try to use memory mapping instead of holding all training vectors in memory

[rom1504]

might also be done for the index itself

would unlock training with more points and building larger indices with a lower memory

[rom1504]

https://stackoverflow.com/a/16867494 adding a memory mapping option to read local embeddings could do it (that option would need to be propagated up to the cli, so things stay fast for high ram availability)

[rom1504]

https://github.com/facebookresearch/faiss/issues/2047 should work probably worth a try

[rom1504]

this could probably significantly improve the performance of indices, but makes sense to do it only after #5 in order to know the performance