zylon-ai
Update ingest.py
How do you justify this change? It will certainly make the ingested database much smaller and reduce the RAM footprint, but the size of the chunks is also reflected in the similarity nearest neighbors used to feed the LLM through the prompt. Making them that large will mean that you will not be able to pass more than one near neighbor to the LLM for the prompt to operate on and generate a result. Because only a single item is passed to the LLM, the quality of the result will suffer and may be out in left field more often than not. Our models have pretty small working contexts (context windows). The Open AI contexts are huge by comparison and permit much larger chunking than we can support reasonably. I suggested increasing the chunk size to 1000 a couple of days ago and I am not certain that wasn't too much.
What was the window size of the gola model
On Wed, 24 May 2023, 7:40 am johnbrisbin, @.***> wrote:
How do you justify this change? It will certainly make the ingested database much smaller and reduce the RAM footprint, but the size of the chunks is also reflected in the similarity nearest neighbors used to feed the LLM through the prompt. Making them that large will mean that you will not be able to pass more than one near neighbor to the LLM for the prompt to operate on and generate a result. Because only a single item is passed to the LLM, the quality of the result will suffer and may be out in left field more often than not. Our models have pretty small working contexts (context windows). The Open AI contexts are huge by comparison and permit much larger chunking than we can support reasonably. I suggested increasing the chunk size to 1000 a couple of days ago and I am not certain that wasn't too much.
— Reply to this email directly, view it on GitHub https://github.com/imartinez/privateGPT/pull/421#issuecomment-1560360837, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOSXNHGUMG42XEMKJDP2RTTXHVU23ANCNFSM6AAAAAAYL2B6DI . You are receiving this because you authored the thread.Message ID: @.***>
I don't think increasing that much the chunk size would be beneficial as a default value, given it'd make the prompt way larger increasing the response times of the LLM. Thanks for the suggestion.
