langchainjs icon indicating copy to clipboard operation
langchainjs copied to clipboard

feat: add Faiss as VectorStore provider

Open ewfian opened this issue 2 years ago • 1 comments

Add support for Faiss and also implemented a loadFromPython method, which can read VectorStore saved from Python's implementation.

TODO:

  • [X] Improve the pickle parser and fully implement Pickle protocol version 4. Extract pickle.js as an NPM package and then import it. The package will be here pickleparser.
  • [x] Port id to uuid mapping from Python implementation.
  • [x] Save mapping into JSON file.
  • [x] Add examples.
    • [x] Create a new index from texts.
    • [x] Create a new index from a loader.
    • [x] Save an index to file and load it again.
    • [x] Load the saved file from Python.
  • [x] Add test cases.
    • [x] Unit Test: fromTexts + addVectors.
    • [x] Unit Test: fromDocuments + addVectors.
    • [x] Unit Test: exceptions.
    • [x] Integration Test: save and load.
    • [x] Integration Test: Load from Python.
    • [x] Integration Test: Add documents after loaded from Python.
    • [x] Integration Test: Save and re-load after loaded from Python.
  • [x] Add documentation.
    • [x] Create pages.
    • [x] Add basic information.
    • [x] Add example for how to read from Python.

ewfian avatar Apr 09 '23 01:04 ewfian

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
langchainjs-docs ✅ Ready (Inspect) Visit Preview May 18, 2023 4:53pm

vercel[bot] avatar Apr 09 '23 01:04 vercel[bot]

Thanks a lot for this @ewfian this is a great PR, sorry for the delay in getting this in

nfcampos avatar May 18 '23 16:05 nfcampos