autogen
autogen copied to clipboard
microsoft
Will retrievechat be supported in pyautogen >= 0.2?
In the docs, it is written:
pyautogen<0.2 supports retrieval-augmented generation tasks such as question answering and code generation with RAG agents.
This makes me wonder if retrievechat will no longer be supported in pyautogen >= 0.2. If so, will a similar functionality be introduced to support RAG use cases?
The version limit is removed now from the documentation. v0.2 does support RAG. cc @thinkall
Hi @hsm207 , have you tried RAG with pyautogen v0.2? The docs have been updated, please let me know if you have encountered any issues. Thanks.
@thinkall I'm running the retrievechat notebook using pyautogen 0.2.2.
In this cell:
I get the error:
InvalidDimensionException Traceback (most recent call last)
Cell In[6], [line 9](vscode-notebook-cell:?execution_count=6&line=9)
[4](vscode-notebook-cell:?execution_count=6&line=4) # given a problem, we use the ragproxyagent to generate a prompt to be sent to the assistant as the initial message.
[5](vscode-notebook-cell:?execution_count=6&line=5) # the assistant receives the message and generates a response. The response will be sent back to the ragproxyagent for processing.
[6](vscode-notebook-cell:?execution_count=6&line=6) # The conversation continues until the termination condition is met, in RetrieveChat, the termination condition when no human-in-loop is no code block detected.
[7](vscode-notebook-cell:?execution_count=6&line=7) # With human-in-loop, the conversation will continue until the user says "exit".
[8](vscode-notebook-cell:?execution_count=6&line=8) code_problem = "How can I use FLAML to perform a classification task and use spark to do parallel training. Train 30 seconds and force cancel jobs if time limit is reached."
----> [9](vscode-notebook-cell:?execution_count=6&line=9) ragproxyagent.initiate_chat(assistant, problem=code_problem, search_string="spark") # search_string is used as an extra filter for the embeddings search, in this case, we only want to search documents that contain "spark".
File [/workspaces/autogen/autogen/agentchat/conversable_agent.py:556](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/conversable_agent.py:556), in ConversableAgent.initiate_chat(self, recipient, clear_history, silent, **context)
[542](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/conversable_agent.py:542) """Initiate a chat with the recipient agent.
[543](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/conversable_agent.py:543)
[544](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/conversable_agent.py:544) Reset the consecutive auto reply counter.
(...)
[553](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/conversable_agent.py:553) "message" needs to be provided if the `generate_init_message` method is not overridden.
[554](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/conversable_agent.py:554) """
[555](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/conversable_agent.py:555) self._prepare_chat(recipient, clear_history)
--> [556](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/conversable_agent.py:556) self.send(self.generate_init_message(**context), recipient, silent=silent)
File [/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:424](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:424), in RetrieveUserProxyAgent.generate_init_message(self, problem, n_results, search_string)
[413](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:413) """Generate an initial message with the given problem and prompt.
[414](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:414)
[415](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:415) Args:
(...)
[421](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:421) str: the generated prompt ready to be sent to the assistant agent.
[422](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:422) """
[423](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:423) self._reset()
--> [424](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:424) self.retrieve_docs(problem, n_results, search_string)
[425](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:425) self.problem = problem
[426](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:426) self.n_results = n_results
File [/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:382](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:382), in RetrieveUserProxyAgent.retrieve_docs(self, problem, n_results, search_string)
[380](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:380) if not self._collection or not self._get_or_create:
[381](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:381) print("Trying to create collection.")
--> [382](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:382) self._client = create_vector_db_from_dir(
[383](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:383) dir_path=self._docs_path,
[384](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:384) max_tokens=self._chunk_token_size,
[385](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:385) client=self._client,
[386](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:386) collection_name=self._collection_name,
[387](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:387) chunk_mode=self._chunk_mode,
[388](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:388) must_break_at_empty_line=self._must_break_at_empty_line,
[389](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:389) embedding_model=self._embedding_model,
[390](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:390) get_or_create=self._get_or_create,
[391](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:391) embedding_function=self._embedding_function,
[392](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:392) custom_text_split_function=self.custom_text_split_function,
[393](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:393) custom_text_types=self._custom_text_types,
[394](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:394) recursive=self._recursive,
[395](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:395) )
[396](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:396) self._collection = True
[397](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/agentchat/contrib/retrieve_user_proxy_agent.py:397) self._get_or_create = True
File [/workspaces/autogen/autogen/retrieve_utils.py:315](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/retrieve_utils.py:315), in create_vector_db_from_dir(dir_path, max_tokens, client, db_path, collection_name, get_or_create, chunk_mode, must_break_at_empty_line, embedding_model, embedding_function, custom_text_split_function, custom_text_types, recursive)
[313](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/retrieve_utils.py:313) for i in range(0, len(chunks), min(40000, len(chunks))):
[314](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/retrieve_utils.py:314) end_idx = i + min(40000, len(chunks) - i)
--> [315](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/retrieve_utils.py:315) collection.upsert(
[316](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/retrieve_utils.py:316) documents=chunks[i:end_idx],
[317](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/retrieve_utils.py:317) ids=[f"doc_{j}" for j in range(i, end_idx)], # unique for each doc
[318](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/retrieve_utils.py:318) )
[319](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/retrieve_utils.py:319) except ValueError as e:
[320](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/autogen/retrieve_utils.py:320) logger.warning(f"{e}")
File [~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:487](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:487), in Collection.upsert(self, ids, embeddings, metadatas, documents, images, uris)
[484](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:484) else:
[485](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:485) embeddings = self._embed(input=images)
--> [487](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:487) self._client._upsert(
[488](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:488) collection_id=self.id,
[489](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:489) ids=ids,
[490](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:490) embeddings=embeddings,
[491](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:491) metadatas=metadatas,
[492](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:492) documents=documents,
[493](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:493) uris=uris,
[494](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/models/Collection.py:494) )
File [~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:127](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:127), in trace_method.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
[125](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:125) global tracer, granularity
[126](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:126) if trace_granularity < granularity:
--> [127](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:127) return f(*args, **kwargs)
[128](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:128) if not tracer:
[129](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:129) return f(*args, **kwargs)
File [~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:461](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:461), in SegmentAPI._upsert(self, collection_id, ids, embeddings, metadatas, documents, uris)
[451](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:451) records_to_submit = []
[452](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:452) for r in _records(
[453](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:453) t.Operation.UPSERT,
[454](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:454) ids=ids,
(...)
[459](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:459) uris=uris,
[460](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:460) ):
--> [461](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:461) self._validate_embedding_record(coll, r)
[462](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:462) records_to_submit.append(r)
[463](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:463) self._producer.submit_embeddings(coll["topic"], records_to_submit)
File [~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:127](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:127), in trace_method.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
[125](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:125) global tracer, granularity
[126](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:126) if trace_granularity < granularity:
--> [127](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:127) return f(*args, **kwargs)
[128](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:128) if not tracer:
[129](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:129) return f(*args, **kwargs)
File [~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:799](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:799), in SegmentAPI._validate_embedding_record(self, collection, record)
[797](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:797) add_attributes_to_current_span({"collection_id": str(collection["id"])})
[798](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:798) if record["embedding"]:
--> [799](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:799) self._validate_dimension(collection, len(record["embedding"]), update=True)
File [~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:127](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:127), in trace_method.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
[125](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:125) global tracer, granularity
[126](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:126) if trace_granularity < granularity:
--> [127](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:127) return f(*args, **kwargs)
[128](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:128) if not tracer:
[129](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/telemetry/opentelemetry/__init__.py:129) return f(*args, **kwargs)
File [~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:814](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:814), in SegmentAPI._validate_dimension(self, collection, dim, update)
[812](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:812) self._collection_cache[id]["dimension"] = dim
[813](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:813) elif collection["dimension"] != dim:
--> [814](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:814) raise InvalidDimensionException(
[815](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:815) f"Embedding dimension {dim} does not match collection dimensionality {collection['dimension']}"
[816](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:816) )
[817](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:817) else:
[818](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f686f6d652f7562756e74752f6175746f67656e222c226c6f63616c446f636b6572223a66616c73652c22636f6e66696746696c65223a7b22246d6964223a312c2270617468223a222f686f6d652f7562756e74752f6175746f67656e2f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a227673636f64652d66696c65486f7374227d7d-0040ssh-002dremote-002bmsh-002dexperiments.vscode-resource.vscode-cdn.net/workspaces/autogen/notebook/~/.local/lib/python3.10/site-packages/chromadb/api/segment.py:818) return
InvalidDimensionException: Embedding dimension 768 does not match collection dimensionality 384
@hsm207 , could you remove the existing database and rerun it? The error indicates the embedding function used for creating the vector db and that for searching are different.
@thinkall how do i remove the existing database (not familiar with chroma)?
Simply remove the path /tmp/chromadb would do the trick.
"client": chromadb.PersistentClient(path="/tmp/chromadb"),
File ~/opt/anaconda3/envs/autogen/lib/python3.11/site-packages/chromadb/utils/embedding_functions.py:133, in OpenAIEmbeddingFunction.__call__(self, texts)
130 texts = [t.replace("\n", " ") for t in texts]
132 # Call the OpenAI Embedding API
--> 133 embeddings = self._client.create(input=texts, engine=self._model_name)["data"]
135 # Sort resulting embeddings by index
136 sorted_embeddings = sorted(embeddings, key=lambda e: e["index"]) # type: ignore
File ~/opt/anaconda3/envs/autogen/lib/python3.11/site-packages/openai/lib/_old_api.py:39, in APIRemovedInV1Proxy.__call__(self, *_args, **_kwargs)
38 def __call__(self, *_args: Any, **_kwargs: Any) -> Any:
---> 39 raise APIRemovedInV1(symbol=self._symbol)
APIRemovedInV1:
You tried to access openai.Embedding, but this is no longer supported in openai>=1.0.0 - see the README at https://github.com/openai/openai-python for the API.
You can run `openai migrate` to automatically upgrade your codebase to use the 1.0.0 interface.
Alternatively, you can pin your installation to the old version, e.g. `pip install openai==0.28`
A detailed migration guide is available here: https://github.com/openai/openai-python/discussions/742
OpenAI embeddings are not working because of a major change in the library
Realised this was coming from chromadb so upgraded to chromadb from git which fixed it:
pip install git+https://github.com/chroma-core/chroma.git