Madeesh Kannan
Madeesh Kannan
Since upgrading to Windows 10 2004, AutoNightMode periodically prompts me to confirm moving the automatically generated _Custom.theme_ file to the recycle bin. Ideally, this file should be permanently deleted without...
## Description This PR introduces the following changes: - Add type checks to ensure unexpected input raises a `TypeError`. - Add `items()` method to iterate over hashes and strings in...
Depends on https://github.com/deepset-ai/haystack/issues/7022. Wrap `LLMEvaluator` to provide a component that calculates the "Context relevance" based on the following inputs: - Questions - Contexts This component is meant to be plug-n-play,...
Depends on https://github.com/deepset-ai/haystack/issues/7022. Wrap `LLMEvaluator` to provide a component that calculates the "Faithfulness" or "Factual accuracy" based on the following inputs: - Questions - Contexts - Responses This component is...
Implement a `LLMEvaluator` component with the following parameters: - `api` - API to use (openai, cohere, etc) - `api_key` - API secret - `inputs` - tuple of expected inputs -...
Now that we have integrations for third party LLM eval frameworks, we need to add support for a handful of LLM-based metrics that we officially support as part of core....
Currently, components with `warm_up` handle non-warmed up states inconsistently. Some call `warm_up` inside of `run`, other raise an error. We should be consistent here. One approach would be to expand...
Follow the same procedure seen in https://github.com/deepset-ai/haystack/pull/6887. ~This will require a new beta release of `haystack-ai`. Once that's out, pin its minimum version to the integration and start porting the...
We currently have components ([OpenAI generator](https://github.com/deepset-ai/haystack/blob/c47b82c54f313cf269ab0c6d4989e6857fdb082e/haystack/components/generators/openai.py#L102), [Gemini integration](https://github.com/deepset-ai/haystack-core-integrations/blob/f8efcfe3126fd4066e708872386c06716f5415a4/integrations/google_ai/src/google_ai_haystack/generators/gemini.py#L103)) that accept an API key as an init parameter. However, for good reason, we do not serialize this key to disk when...
Implement scaffolding code that: - Accepts... - The evaluated pipeline, i.e., the pipeline whose output is to be evaluated. - A set of inputs for the above pipeline. - The...