deepeval icon indicating copy to clipboard operation
deepeval copied to clipboard

Corrupted cache can make it impossible to run deepeval

Open prescod opened this issue 1 month ago • 0 comments

Describe the bug Cache can get corrupted and block execution.

To Reproduce

touch .temp-deepeval-cache.json

Then run a program with caching turned on.

The bug can also be sort of seen in the code.

The function signature for get_cached_test_run is

    def get_cached_test_run(
        self, from_temp: bool = False
    ) -> Union[CachedTestRun, None]:

And yet the way it is used is:

        cached_test_run = self.get_cached_test_run(from_temp=to_temp)
        cached_test_run.test_cases_lookup_map[test_case_cache_key] = (
            new_cache_test_case
        )

There is no handling for the None case, which results in an exception.

  File "/Users/paul.prescod/.venvs/evalu8/lib/python3.10/site-packages/deepeval/evaluate.py", line 348, in a_execute_test_cases
    test_run_cache_manager.cache_test_case(
  File "/Users/paul.prescod/.venvs/evalu8/lib/python3.10/site-packages/deepeval/test_run/cache.py", line 136, in cache_test_case
    cached_test_run.test_cases_lookup_map[test_case_cache_key] = (
AttributeError: 'NoneType' object has no attribute 'test_cases_lookup_map'

https://github.com/confident-ai/deepeval/blob/004e1cb42d11d9de039a0db047423f9576f70800/deepeval/test_run/cache.py#L136

If cached_test_run can be None, the code should handle that.

Expected behavior Replace the cache if it is missing or corrupted.

Deepeval version 0.21.42

prescod avatar May 15 '24 00:05 prescod