llama_index
llama_index copied to clipboard
run-llama
Responses have cut off text
Using GPTSimpleVectorIndex with an example call:
response = index.query("How can a co-owner object to a meeting?", similarity_top_k=3, verbose=True)
I get text that is truncated:
e.g.
A co-owner can object to a meeting by applying to the court to annul or amend the decision of the general meeting if the decision is biased, taken with intent to injure the co-owners or in contempt of their rights, or if an error was made in counting the votes. The action must be instituted within 90 days after the meeting or it will be forfeited. If the action is futile or vexatious, the court may condemn the plaintiff to pay damages. Additionally, if the co-owners are prevented from acting as a majority or in the specified proportion owing to an impediment or the systematic opposition of some of them, the court may, on the application of a co-owner, make any order it sees fit in the circumstances. Furthermore, any co-owner may cause a question to be placed on the agenda within five days of receiving notice of the meeting, and the board of directors must give written notice to the co-owners of the questions newly placed on the agenda before the meeting is held. Finally, a meeting may be held by the use of a means which allows all those participating to communicate directly with each other. Co-owners who participate in a meeting by the use of such a means may vote by any means enabling votes to
Total LLM tokens were small: Total LLM token usage: 1901 tokens
Is there a way to specify the max tokens of the response to make it larger so it doesn't get truncated?
you have to increase the max_tokens in the underlying LLM class from langchain (see https://gpt-index.readthedocs.io/en/latest/how_to/custom_llms.html)
Here is a snippet I used to solve for others...
from gpt_index import LLMPredictor, GPTSimpleVectorIndex
from langchain.llms import OpenAI
llm_predictor = LLMPredictor(llm=OpenAI(max_tokens=512, model_name="text-davinci-003"))
index = GPTSimpleVectorIndex.load_from_disk(path_to_index, llm_predictor=llm_predictor)
response = index.query("Insert prompt here...", verbose=False)
display(Markdown(f"{response}"))
where max_tokens=512 can be adjusted to your liking
