Enable Criteria Evaluation in Langflow

[sampratha28]

I want to make use of criteria evaluation in langflow to verify if the model's response is grounded to the vector store or not. I see that there is already code available in the langchain repository. How to enable it?

Also note that i am creating a custom LLM component

[dosubot[bot]]

🤖

Hello @sampratha28! 👋

I'm Dosu, a handy bot here to assist you with your queries and issues regarding langflow. I'm here to help you navigate through your bug fixes, answer your questions, and guide you on your journey to being a contributor.

While we're waiting for a human maintainer to step in, feel free to ask me anything! We're in this together.

Happy coding! 🚀

Based on the current implementation of the SelfQueryRetriever.from_llm method in the VectaraSelfQueryRetriverComponent class, there is no criteria parameter available. This method currently accepts the following parameters: llm, vectorstore, document_content_description, metadata_field_obj, and verbose.

If you want to add a criteria evaluation to verify if the model's response is grounded to the vector store, you would need to modify the SelfQueryRetriever.from_llm method to accept an additional criteria parameter. This criteria parameter should be a function that takes in the model's response and returns a boolean indicating whether the response is grounded to the vector store.

Here's an example of how you can modify the SelfQueryRetriever.from_llm method:

@classmethod
def from_llm(
    cls,
    llm: BaseLanguageModel,
    vectorstore: VectorStore,
    document_content_description: str,
    metadata_field_info: List[AttributeInfo],
    verbose: bool = False,
    criteria: Callable[[Any], bool] = None  # Add this line
) -> 'SelfQueryRetriever':
    # existing code...

And then in the build method of the VectaraSelfQueryRetriverComponent class, you can pass the criteria function:

return SelfQueryRetriever.from_llm(
    llm,
    vectorstore, 
    document_content_description, 
    metadata_field_obj, 
    verbose=True,
    criteria=criteria  # Add this line
)

Please note that you need to define the criteria function according to your specific requirements. This function should take in the model's response and return a boolean indicating whether the response is grounded to the vector store.

However, please be aware that this is a significant change and might affect other parts of the codebase that use the SelfQueryRetriever.from_llm method. It's recommended to thoroughly test your changes to ensure that they don't introduce any unexpected behavior.

You can refer to the VectaraSelfQueryRetriverComponent class for more details.

I hope this helps! If you have any other questions, feel free to ask.

Best, Dosu

Sources

• src/backend/langflow/components/retrievers/VectaraSelfQueryRetriver.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.