Allow caching when evaluateOptions.repeat is greater than 1

[crivano]

My test cases always use multiple repetitions to check if the prompt is stable. Unfortunately, PromptFoo never caches responses when repeat > 1.

It would be great if cache works in this situation.

[mldangelo]

Thank you for your suggestion, @crivano. This is a great idea and has been requested by other users as well (including myself). Implementing this feature involves making modifications across multiple parts of the codebase, including:

1. src/cache.ts:

   • Modifying the fetchWithCache function to include the repeat number in the cache key.

2. Provider-specific files in src/providers/:

   • Updating cache key generation in each provider that implements its own caching.
   • Examples include:
     • src/providers/anthropic.ts
     • src/providers/openai.ts
     • src/providers/azureopenai.ts
     • (and any other providers with custom caching - there are 26 instances of cacheKey)

3. src/evaluator.ts:

   • Updating the evaluate method to pass the repeat number to provider calls.

4. Test files in the test/ directory:

   • Updating existing tests and adding new ones to cover the new caching behavior with repetitions.

5. Documentation files:

   • Updating documentation to reflect the new caching capabilities.

I will add this to the backlog, but we likely won't get to it for a while. We would welcome a PR if you or someone else from the community is interested in helping out with this. Thank you again for your valuable feedback!