haystack
haystack copied to clipboard
Make use of the `seed` parameter in the LLM-based evaluators using the OpenAI API
Is your feature request related to a problem? Please describe. The LLM-based evaluators are non-deterministic, and this can be an issue in replicating and comparing evaluation results.
Describe the solution you'd like
One possible solution is to use the seed
parameter when using the OpenAI-based LLMs: https://platform.openai.com/docs/api-reference/chat/create
-
seed
: integer or null - This feature is in Beta. If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result.