text-generation-webui Superboogav2 occasionally fails in chat: Cannot return the results in a contiguous 2D array

Describe the bug

While Superboogav2 is active, messages in chat mode occasionally result in the following error, preventing a reply from being generated. A quick search online suggests this may be an issue with chromaDB itself (see https://github.com/langchain-ai/langchain/issues/3665 for an example), but we should probably handle this a little better here, too, by either making the ef/M values configurable, adding in some means of determining suitable ones automatically, or simply handling the error gracefully (skipping the DB and just processing the prompt as normal).

Is there an existing issue for this?

[X] I have searched the existing issues

Reproduction

Ensure superboogav2 is enabled.
Chat until error appears.

Note: I've noticed this issue occur with both very long chats, and very short ones; if I discover more detailed reproduction steps, I will update here.

Screenshot

No response

Logs

08:18:27-930545 INFO     Successfully deleted 49 records from chromaDB.
08:18:29-395236 INFO     Adding 116 new embeddings.
Traceback (most recent call last):
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\gradio\queueing.py", line 407, in call_prediction
    output = await route_utils.call_process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\gradio\route_utils.py", line 226, in call_process_api
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\gradio\blocks.py", line 1550, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\gradio\blocks.py", line 1199, in call_function
    prediction = await utils.async_iteration(iterator)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\gradio\utils.py", line 519, in async_iteration
    return await iterator.__anext__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\gradio\utils.py", line 512, in __anext__
    return await anyio.to_thread.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\gradio\utils.py", line 495, in run_sync_iterator_async
    return next(iterator)
           ^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\gradio\utils.py", line 649, in gen_wrapper
    yield from f(*args, **kwargs)
  File "C:\Source\Repos\text-generation-webui\modules\chat.py", line 365, in generate_chat_reply_wrapper
    for i, history in enumerate(generate_chat_reply(text, state, regenerate, _continue, loading_message=True, for_ui=True)):
  File "C:\Source\Repos\text-generation-webui\modules\chat.py", line 333, in generate_chat_reply
    for history in chatbot_wrapper(text, state, regenerate=regenerate, _continue=_continue, loading_message=loading_message, for_ui=for_ui):
  File "C:\Source\Repos\text-generation-webui\modules\chat.py", line 270, in chatbot_wrapper
    prompt = apply_extensions('custom_generate_chat_prompt', text, state, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\modules\extensions.py", line 229, in apply_extensions
    return EXTENSION_MAP[typ](*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\modules\extensions.py", line 105, in _apply_custom_generate_chat_prompt
    return extension.custom_generate_chat_prompt(text, state, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\extensions\superboogav2\script.py", line 167, in custom_generate_chat_prompt
    return custom_generate_chat_prompt_internal(user_input, state, collector, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\extensions\superboogav2\chat_handler.py", line 114, in custom_generate_chat_prompt_internal
    results = collector.get_sorted_by_dist(user_input, n_results=parameters.get_chunk_count(), max_token_count=int(parameters.get_max_token_count()))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\extensions\superboogav2\chromadb.py", line 334, in get_sorted_by_dist
    documents, _, distances = self._get_documents_ids_distances(search_strings, n_results)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\extensions\superboogav2\chromadb.py", line 264, in _get_documents_ids_distances
    result = self.collection.query(query_texts=search_string, n_results=math.ceil(n_results / len(search_strings)), include=['distances'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\chromadb\api\models\Collection.py", line 203, in query
    return self._client._query(
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\chromadb\api\local.py", line 247, in _query
    uuids, distances = self._db.get_nearest_neighbors(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\chromadb\db\clickhouse.py", line 521, in get_nearest_neighbors
    uuids, distances = index.get_nearest_neighbors(embeddings, n_results, ids)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Source\Repos\text-generation-webui\installer_files\env\Lib\site-packages\chromadb\db\index\hnswlib.py", line 248, in get_nearest_neighbors
    database_labels, distances = self._index.knn_query(query, k=k, filter=filter_function)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Cannot return the results in a contiguous 2D array. Probably ef or M is too small

System Info

OS: Windows 11
GPU: Nvidia GeForce RTX 2060
Branch: Dev
Commit: cbf6f9e

Dec 31 '23 21:12 TheInvisibleMage

I'm getting the same error. Is there a certain version of chromadb we should have installed, or any settings we need to tweak?

For me, this intermittantly happens when using larger prompt lengths. I'm curious if I also need to adjust the superbooga settings to accommodate?

OS: Ubuntu 22.04 GPU: Nvidia Quadro RTX 5000 Branch: Main Commit: 27decbf

Feb 07 '24 15:02 odysseyalive

In searching through other discussions on this topic, it was mentioned that this is an issue with ChromaDB. There was a suggestion to change the k top value to 10.

suggested parameter values

Feb 08 '24 01:02 odysseyalive

This issue has been closed due to inactivity for 2 months. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.