Improve caching mechanism for prompt generation and evaluation

[ZenomHunter123]

Changes:

• Refactor the existing caching mechanism in evals/utils.py to utilize a more efficient and flexible data structure, such as an LRU cache, to store prompt and evaluation results.
• Introduce a new cache eviction policy to ensure that only the most relevant and frequently used data is retained in the cache, optimizing memory usage.
• Update the cache invalidation logic to handle edge cases where data may not be properly updated, causing inconsistencies when running evaluations.
• Add unit tests to verify the functionality and performance of the new caching mechanism.
• Benefits:

The proposed changes will provide the following benefits:

Reduced API calls: By caching prompt and evaluation results more efficiently, we can minimize the number of API calls required during the evaluation process, thus reducing costs and speeding up the evaluation process. Improved resource optimization: By using an LRU cache with an appropriate eviction policy, we can ensure that the cache is always holding the most relevant data while optimizing memory usage. Increased evaluation consistency: Properly handling edge cases in cache invalidation will result in more consistent evaluation results and minimize any discrepancies that may occur due to stale data in the cache. Easier maintenance and extensibility: Refactoring the caching mechanism and adding comprehensive unit tests will make it easier for future contributors to understand and extend the caching functionality, ensuring the continued success of the Evals framework.

I kindly request that you review this pull request and consider granting me GPT-4 access upon acceptance. My email address associated with this contribution is: [email protected]