llama_index icon indicating copy to clipboard operation
llama_index copied to clipboard

Published 20 hours ago • verified publisher icon run-llama

[Question]: error when I run the llmrerank

Open lambda7xx opened this issue 1 year ago • 7 comments
trafficstars

Question Validation

  • [X] I have searched both the documentation and discord for an answer.

Question

I run the llm reranker

My error is that

ValidationError                           Traceback (most recent call last)
Cell In[8], [line 5](vscode-notebook-cell:?execution_count=8&line=5)
      [1](vscode-notebook-cell:?execution_count=8&line=1) new_nodes = get_retrieved_nodes(
      [2](vscode-notebook-cell:?execution_count=8&line=2)     "What is Lyft's response to COVID-19?", vector_top_k=5, with_reranker=False
      [3](vscode-notebook-cell:?execution_count=8&line=3) )
----> [5](vscode-notebook-cell:?execution_count=8&line=5) visualize_retrieved_nodes(new_nodes)

Cell In[7], [line 43](vscode-notebook-cell:?execution_count=7&line=43)
     [41](vscode-notebook-cell:?execution_count=7&line=41) for node in nodes:
     [42](vscode-notebook-cell:?execution_count=7&line=42)     node = deepcopy(node)
---> [43](vscode-notebook-cell:?execution_count=7&line=43)     node.node.metadata = None
     [44](vscode-notebook-cell:?execution_count=7&line=44)     node_text = node.node.get_text()
     [45](vscode-notebook-cell:?execution_count=7&line=45)     node_text = node_text.replace("\n", " ")

File [/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/v1/main.py:384](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/v1/main.py:384), in BaseModel.__setattr__(self, name, value)
    [382](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/v1/main.py:382) value, error_ = known_field.validate(value, dict_without_original_value, loc=name, cls=self.__class__)
    [383](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/v1/main.py:383) if error_:
--> [384](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/v1/main.py:384)     raise ValidationError([error_], self.__class__)
    [385](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/v1/main.py:385) else:
    [386](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/v1/main.py:386)     new_values[name] = value

ValidationError: 1 validation error for TextNode
metadata
  none is not an allowed value (type=type_error.none.not_allowed)

how to fix? My llama-index version is 0.10.24

lambda7xx avatar Mar 27 '24 08:03 lambda7xx

Hey there, @lambda7xx! 😊 Great to see you back with us. How's everything going on your end?

The error you're encountering is a ValidationError from Pydantic, a Python library for data validation. This error is being raised because the metadata attribute of the node object is being set to None, which is not an allowed value according to the data model for the node object.

The line of code causing the error is node.node.metadata = None. This line is attempting to set the metadata attribute of the node object to None. However, the data model for the node object does not allow None as a valid value for the metadata attribute, hence the ValidationError.

To fix this error, you need to ensure that the metadata attribute is always set to a valid value. If you want to clear the metadata attribute, you could set it to an empty dictionary instead of None. Here's how you can do it:

node.node.metadata = {}

This will set the metadata attribute to an empty dictionary, which should be a valid value according to the data model for the node object.

Please try this solution and let me know if it resolves your issue. If you continue to encounter problems, please provide more details about your use case and the context in which you're using the LLMReranker class, and I'll be happy to help further.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

dosubot[bot] avatar Mar 27 '24 08:03 dosubot[bot]

another problem.

new_nodes = get_retrieved_nodes(
    "What is Lyft's response to COVID-19?",
    vector_top_k=10,
    reranker_top_n=5,
    with_reranker=True,
)

visualize_retrieved_nodes(new_nodes)

the error is

IndexError                                Traceback (most recent call last)
Cell In[12], [line 1](vscode-notebook-cell:?execution_count=12&line=1)
----> [1](vscode-notebook-cell:?execution_count=12&line=1) new_nodes = get_retrieved_nodes(
      [2](vscode-notebook-cell:?execution_count=12&line=2)     "What is Lyft's response to COVID-19?",
      [3](vscode-notebook-cell:?execution_count=12&line=3)     vector_top_k=10,
      [4](vscode-notebook-cell:?execution_count=12&line=4)     reranker_top_n=5,
      [5](vscode-notebook-cell:?execution_count=12&line=5)     with_reranker=True,
      [6](vscode-notebook-cell:?execution_count=12&line=6) )
      [8](vscode-notebook-cell:?execution_count=12&line=8) visualize_retrieved_nodes(new_nodes)

Cell In[9], [line 28](vscode-notebook-cell:?execution_count=9&line=28)
     [22](vscode-notebook-cell:?execution_count=9&line=22) if with_reranker:
     [23](vscode-notebook-cell:?execution_count=9&line=23)     # configure reranker
     [24](vscode-notebook-cell:?execution_count=9&line=24)     reranker = LLMRerank(
     [25](vscode-notebook-cell:?execution_count=9&line=25)         choice_batch_size=5,
     [26](vscode-notebook-cell:?execution_count=9&line=26)         top_n=reranker_top_n,
     [27](vscode-notebook-cell:?execution_count=9&line=27)     )
---> [28](vscode-notebook-cell:?execution_count=9&line=28)     retrieved_nodes = reranker.postprocess_nodes(
     [29](vscode-notebook-cell:?execution_count=9&line=29)         retrieved_nodes, query_bundle
     [30](vscode-notebook-cell:?execution_count=9&line=30)     )
     [32](vscode-notebook-cell:?execution_count=9&line=32) return retrieved_nodes

File [/opt/conda/envs/pytorch/lib/python3.10/site-packages/llama_index/core/postprocessor/types.py:55](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/llama_index/core/postprocessor/types.py:55), in BaseNodePostprocessor.postprocess_nodes(self, nodes, query_bundle, query_str)
     [53](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/llama_index/core/postprocessor/types.py:53) else:
...
--> [109](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/llama_index/core/indices/utils.py:109)     _answer_relevance = re.findall(r"\d+", line_tokens[1].split(":")[1].strip())[0]
    [110](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/llama_index/core/indices/utils.py:110)     answer_relevances.append(float(_answer_relevance))
    [111](https://vscode-remote+ssh-002dremote-002bllama.vscode-resource.vscode-cdn.net/opt/conda/envs/pytorch/lib/python3.10/site-packages/llama_index/core/indices/utils.py:111) return answer_nums, answer_relevances

IndexError: list index out of range

lambda7xx avatar Mar 27 '24 08:03 lambda7xx

@dosubot

lambda7xx avatar Mar 27 '24 08:03 lambda7xx

The IndexError you're encountering in the llama_index/core/indices/utils.py file is likely due to the input data not matching the expected format. Specifically, the code attempts to access an index in a list that doesn't exist, which suggests that the input data's structure is different from what the code anticipates.

This issue often arises when the input data doesn't follow the expected format, particularly in the line where the code tries to parse the answer number. The expected format seems to involve a line of text where the first token, when split by ":", has at least two elements. If the input data deviates from this format, it could lead to the IndexError.

A similar issue was addressed in the LlamaIndex repository, where a user faced a problem with Chinese queries. The solution involved customizing the parsing method to handle the input data format correctly. You might find this approach useful for your case. Here's how you can implement a custom parsing function based on that solution:

from llama_index.core.indices.utils import default_parse_choice_select_answer_fn

def custom_parse_choice_select_answer_fn(answer: str, num_choices: int):
    print('====>>', answer)
    _answer = answer.split('Explanation')[0]
    return default_parse_choice_select_answer_fn(_answer, num_choices)

reranker = LLMRerank(top_n=5, llm=llm, parse_choice_select_answer_fn=custom_parse_choice_select_answer_fn)

chat_engine = index.as_chat_engine(node_postprocessors=[postprocessor, reranker], ...)

This custom function intercepts the answer content before the "Explanation" part, potentially avoiding the IndexError. Try adapting this solution to your scenario, and it should help resolve the issue. If you encounter further difficulties or have additional questions, feel free to ask.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

dosubot[bot] avatar Mar 27 '24 09:03 dosubot[bot]

thanks, I fix the error.

lambda7xx avatar Mar 27 '24 09:03 lambda7xx

my code is below

import nest_asyncio

nest_asyncio.apply()

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.postprocessor import LLMRerank


from IPython.display import Markdown, display

import llama_index.core 

llama_index.core.set_global_handler("simple")

from llama_index.core import Settings
from llama_index.llms.huggingface import HuggingFaceLLM
from llama_index.core.prompts import PromptTemplate
import torch 
from llama_index.core import VectorStoreIndex
from llama_index.core import  SimpleDirectoryReader

llm = HuggingFaceLLM(
    model_name="mistralai/Mistral-7B-Instruct-v0.1",
    tokenizer_name="mistralai/Mistral-7B-Instruct-v0.1",
    query_wrapper_prompt=PromptTemplate("<s>[INST] {query_str} [/INST] </s>\n"),
    context_window=3900,
    max_new_tokens=256,
    model_kwargs={"torch_dtype": torch.bfloat16},
    # tokenizer_kwargs={},
    generate_kwargs={"temperature": 0.2, "top_k": 5, "top_p": 0.95},
    device_map="auto",
)

# LLM (gpt-3.5-turbo)


Settings.llm = llm

embeddings = ["local:BAAI/bge-small-en-v1.5"]

Settings.embed_model = embeddings[0]

Settings.chunk_overlap = 0
Settings.chunk_size = 128

# load documents
import time
start = time.time()
documents = SimpleDirectoryReader(
    input_files=["./data/10k/lyft_2021.pdf"]
).load_data()
end = time.time()
print(f"duration:{end-start}")

index = VectorStoreIndex.from_documents(
    documents,
)


from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core import QueryBundle
import pandas as pd
from IPython.display import display, HTML
from copy import deepcopy


pd.set_option("display.max_colwidth", 1)

from llama_index.core.indices.utils import default_parse_choice_select_answer_fn

def custom_parse_choice_select_answer_fn(answer: str, num_choices: int):
    print('====>>', answer)
    _answer = answer.split('Explanation')[0]
    return default_parse_choice_select_answer_fn(_answer, num_choices)



def get_retrieved_nodes(
    query_str, vector_top_k=10, reranker_top_n=3, with_reranker=False
):
    query_bundle = QueryBundle(query_str)
    # configure retriever
    retriever = VectorIndexRetriever(
        index=index,
        similarity_top_k=vector_top_k,
    )
    retrieved_nodes = retriever.retrieve(query_bundle)

    if with_reranker:
        # configure reranker
        reranker = LLMRerank(
            choice_batch_size=5,
            top_n=reranker_top_n,
            llm = Settings.llm,
            parse_choice_select_answer_fn=custom_parse_choice_select_answer_fn,
        )
        retrieved_nodes = reranker.postprocess_nodes(
            retrieved_nodes, query_bundle
        )

    return retrieved_nodes


def pretty_print(df):
    return display(HTML(df.to_html().replace("\\n", "<br>")))


def visualize_retrieved_nodes(nodes) -> None:
    result_dicts = []
    for node in nodes:
        node = deepcopy(node)
        node.node.metadata = {}
        node_text = node.node.get_text()
        node_text = node_text.replace("\n", " ")

        result_dict = {"Score": node.score, "Text": node_text}
        result_dicts.append(result_dict)

    pretty_print(pd.DataFrame(result_dicts))



import time 
start = time.time()

new_nodes = get_retrieved_nodes(
    "What is Lyft's response to COVID-19?", vector_top_k=5, with_reranker=False
)


visualize_retrieved_nodes(new_nodes)

end = time.time()

print(f"************ first rerank, duration:{end - start }\n\n")

start = time.time()
new_nodes = get_retrieved_nodes(
    "What is Lyft's response to COVID-19?",
    vector_top_k=40,
    reranker_top_n=5,
    with_reranker=True,
)

the vector_top_k = 40, then I have some error.

Doc: 4,
Traceback (most recent call last):
  File "/home/ubuntu/uw-llama/rerank/llm_reanker.py", line 143, in <module>
    new_nodes = get_retrieved_nodes(
  File "/home/ubuntu/uw-llama/rerank/llm_reanker.py", line 102, in get_retrieved_nodes
    retrieved_nodes = reranker.postprocess_nodes(
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/llama_index/core/postprocessor/types.py", line 55, in postprocess_nodes
    return self._postprocess_nodes(nodes, query_bundle)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/llama_index/core/postprocessor/llm_rerank.py", line 99, in _postprocess_nodes
    raw_choices, relevances = self._parse_choice_select_answer_fn(
  File "/home/ubuntu/uw-llama/rerank/llm_reanker.py", line 79, in custom_parse_choice_select_answer_fn
    return default_parse_choice_select_answer_fn(_answer, num_choices)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/llama_index/core/indices/utils.py", line 109, in default_parse_choice_select_answer_fn
    _answer_relevance = re.findall(r"\d+", line_tokens[1].split(":")[1].strip())[0]
IndexError: list index out of range

how to fix.

lambda7xx avatar Mar 27 '24 12:03 lambda7xx

Also, if I want to run query on the rerank . how?

lambda7xx avatar Mar 27 '24 12:03 lambda7xx