faithfulness_score: nan

[beatG123]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question faithfulness_score: always be nan

Code Examples from agent import * from datasets import Dataset import json import random

文本加载

from langchain_community.document_loaders import WebBaseLoader loader = WebBaseLoader("https://baike.baidu.com/item/%E7%BA%BD%E7%BA%A6/6230") loader.requests_kwargs = {'verify':False} data = loader.load()

print(data)

创建向量索引

from langchain.indexes import VectorstoreIndexCreator index = VectorstoreIndexCreator().from_loaders([loader])

创建qa链

from langchain.chains import RetrievalQA from langchain_community.chat_models import ChatOpenAI llm = ChatOpenAI() qa_chain = RetrievalQA.from_chain_type( llm, retriever=index.vectorstore.as_retriever(), return_source_documents=True,verbose = VERBOSE )

question = "纽约市的名字是怎么得来的?" result = qa_chain({"query": question})

result["result"]

print("========= chain result ==========")

print(result)

result['ground_truths'] = "纽约市的名字“纽约”来源于荷兰战败后将新阿姆斯特丹割让给英国的事件。" from ragas.metrics import faithfulness, answer_relevancy, context_relevancy, context_recall from ragas.langchain.evalchain import RagasEvaluatorChain

make eval chains

eval_chains = { m.name: RagasEvaluatorChain(metric=m) for m in [faithfulness, answer_relevancy, context_relevancy, context_recall] }

evaluate

for name, eval_chain in eval_chains.items(): score_name = f"{name}_score" print(f"{score_name}: {eval_chain(result)[score_name]}")

Additional context

Entering new RetrievalQA chain...

Finished chain. faithfulness_score: nan answer_relevancy_score: 0.8500799716619457 context_relevancy_score: 0.016666666666666666 context_recall_score: 1.0