haystack icon indicating copy to clipboard operation
haystack copied to clipboard

Make use of the `seed` parameter in the LLM-based evaluators using the OpenAI API

Open davidsbatista opened this issue 1 month ago • 0 comments

Is your feature request related to a problem? Please describe. The LLM-based evaluators are non-deterministic, and this can be an issue in replicating and comparing evaluation results.

Describe the solution you'd like One possible solution is to use the seed parameter when using the OpenAI-based LLMs: https://platform.openai.com/docs/api-reference/chat/create

  • seed: integer or null
  • This feature is in Beta. If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result.

davidsbatista avatar May 17 '24 12:05 davidsbatista