[R-230] Adapt prompts "in place" also for cached prompts

[HerrIvan]

Describe the Feature Adapt Prompts "in place", also in case they are retrieved from the cache.

Why is the feature important for you? If you want to generate a testset in a non-english language, the right approach is tofirst adapt all the generation/filtering prompts and then run the generation. However, this works only if you adapt the prompts the first time. In that case, the prompt instances get changed "in place", that means that when the testset.generate accesses the prompts, these prompts are the translated ones. However, if the prompts are retrieved from the cache, they are not changed "in place", which means that when TestsetGenerator.generate accesses the prompts, it uses the original ones in English.

Additional context This is half a feature request, half a bug, since the behavior of the "adapt" function is not consistent: Prompts in the cache are not modified "in place", while instances of prompts not in the cache are modified.

See below in src/ragas/llms/prompt.py, when a prompt is "translated/adapted" the translated values are stored in the prompt instance. On top of that, the instance is returned.

        for i, example in enumerate(grouped_results):
            [...]
            self.examples[i] = example_dict

        self.language = language

        # TODO:Validate the prompt after adaptation

        return self

However, in the same function, whenever there is a hit in the cache the current prompt instance isn't modified (ie, nothing is assigned to the self var:

        if os.path.exists(os.path.join(cache_dir, language, f"{self.name}.json")):
            return self._load(language, self.name, cache_dir)

(The function self._load is a class method which doesn't modify the instance).

I will open a PR to address this issue.

_R-230

[shahules786]

Hey @HerrIvan thanks for pointing this out. Seems like a bug to me.

[jjmachan]

also tagging #890

[jjmachan]

This has been fixed with v0.2 - I know finally 😅 🎉

do checkout the docs here: https://docs.ragas.io/en/stable/howtos/customizations/metrics/_metrics_language_adaptation/ reference here: https://docs.ragas.io/en/stable/references/prompt/#ragas.prompt.PromptMixin

and if you're migrating from v0.1 check out the migration docs here: https://docs.ragas.io/en/stable/howtos/migrations/migrate_from_v01_to_v02

could you check it out and verify if not feel free to comment here and I'll help you out - really sorry again that it tool this while