Hook into Runnable RAG chain

[amitjoy]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question How can I integrate EvaluatorChain with an existing Runnable RAG chain?

Code Examples An example of chain created using LCEL:

    def chain(self) -> Runnable:
        """
        Constructs the retrieval chain for the chat agent.

        :return: A Runnable instance representing the retrieval chain.
        """
        logger.debug("Initializing retrieval from knowledge base chain")

        history_aware_retriever = create_history_aware_retriever(
            llm=self.vertex.model,
            retriever=self.kb_agent.retriever,
            prompt=self.condense_prompt
        )
        document_chain = create_stuff_documents_chain(
            llm=self.vertex.model,
            prompt=self.prompt
        )
        retrieval_chain = create_retrieval_chain(
            retriever=history_aware_retriever,
            combine_docs_chain=document_chain
        )

        def get_session_history(session_id: str) -> BaseChatMessageHistory:
            return self.history_agent.message_history

        return RunnableWithMessageHistory(
            retrieval_chain,
            get_session_history,
            input_messages_key="input",
            history_messages_key="chat_history",
            output_messages_key="answer"
        )

Additional context running on latest versions of Langchain and Ragas

[dosubot[bot]]

To integrate EvaluatorChain with your existing Runnable RAG chain, you need to instantiate the EvaluatorChain with the appropriate metric and then use it within your RAG chain. Below is an example of how you can achieve this:

1. Instantiate the EvaluatorChain:

   from ragas.metrics import SomeSpecificMetric  # Replace with the actual metric you want to use
   from ragas.langchain.evalchain import RagasEvaluatorChain
   
   # Initialize the metric
   metric = SomeSpecificMetric()
   
   # Create the EvaluatorChain
   evaluator_chain = RagasEvaluatorChain(metric=metric)
   

2. Integrate the EvaluatorChain into your RAG chain: Assuming you have an existing RAG chain, you can call the EvaluatorChain within it. Here is a simplified example based on your provided code:

   from langchain.chains import SomeRAGChain  # Replace with your actual RAG chain
   
   def chain(self) -> Runnable:
       """
       Constructs the retrieval chain for the chat agent.
   
       :return: A Runnable instance representing the retrieval chain.
       """
       logger.debug("Initializing retrieval from knowledge base chain")
   
       history_aware_retriever = create_history_aware_retriever(
           llm=self.vertex.model,
           retriever=self.kb_agent.retriever,
           prompt=self.condense_prompt
       )
       document_chain = create_stuff_documents_chain(
           llm=self.vertex.model,
           prompt=self.prompt
       )
       retrieval_chain = create_retrieval_chain(
           retriever=history_aware_retriever,
           combine_docs_chain=document_chain
       )
   
       def get_session_history(session_id: str) -> BaseChatMessageHistory:
           return self.history_agent.message_history
   
       # Integrate the EvaluatorChain
       def evaluate_chain(inputs):
           rag_outputs = retrieval_chain(inputs)
           evaluation_result = evaluator_chain(rag_outputs)
           return evaluation_result
   
       return RunnableWithMessageHistory(
           evaluate_chain,
           get_session_history,
           input_messages_key="input",
           history_messages_key="chat_history",
           output_messages_key="answer"
       )
   

In this example, replace SomeSpecificMetric with the actual metric you want to use and SomeRAGChain with your actual RAG chain class. The evaluate_chain function runs the RAG chain and then evaluates the outputs using the EvaluatorChain.

This integration allows you to evaluate the outputs of your RAG chain using the specified metric within the EvaluatorChain ^[1][2].

To continue talking to Dosu, mention @dosu.

[jjmachan]

@amitjoy where you able to fix this?

[anistark]

Closing this as resolved.

Check EvaluatorChain usage. Docs pending to better reflect this.