Cohere embeddings can't be pickled

[clharman]

Running the following code:

store = FAISS.from_texts(
            chunks["texts"], embeddings_instance, metadatas=chunks["metadatas"]
        )
faiss.write_index(store.index, "index.faiss")
store.index = None
with open("faiss_store.pkl", "wb") as f:
    pickle.dump(store, f)

Returns the following errors:

Error ingesting document: cannot pickle '_queue.SimpleQueue' object
Traceback (most recent call last):
  File "/Users/colin/data/form-filler-demo/ingest.py", line 95, in ingest_pdf
    index_chunks(chunks)
  File "/Users/colin/data/form-filler-demo/ingest.py", line 72, in index_chunks
    pickle.dump(store, f)
TypeError: cannot pickle '_queue.SimpleQueue' object
2023-02-22 13:27:53.225 Uncaught app exception
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)
  File "/Users/colin/data/form-filler-demo/main.py", line 77, in <module>
    store = pickle.load(f)
EOFError: Ran out of input

When using embeddings_instance=CohereEmbeddings(). Works fine with OpenAIEmbeddings(). Planning to look into it when I have time, unless somebody else has experienced this!

[ellisonbg]

Hmmm, I am using both the Cohre and OpenAI embeddings and it works fine. The code I am using the slightly different:

embeddings = CohereEmbeddings()
vectordb = FAISS.from_documents(documents=docs, embedding=embeddings)
vectordb.save_local('./vectordb')

Maybe try the save_local method?

[andrewtryder]

I am running into the same error trying to use CohereEmbeddings().

I think it is related to this issue?

https://github.com/cohere-ai/sandbox-conversant-lib/pull/46

[dosubot[bot]]

Hi, @clharman! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding, the issue you reported is related to pickling a '_queue.SimpleQueue' object when trying to pickle a store object using Cohere embeddings. One potential solution suggested by ellisonbg is to try the save_local method. Another user, andrewtryder, mentioned that they are also experiencing the same error and believe it may be related to a pull request in the repository.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your contribution to the LangChain repository!