ragas icon indicating copy to clipboard operation
ragas copied to clipboard

Fixed metric adaptation issue with the prompt factory

Open ssoima opened this issue 1 year ago • 2 comments

Issue: Runtime metric language adaptation (e.g. adapting to italian and then to german in the same run) does not work.

The reason is that the @dataclass factories in the metrics use singleton objects (which only get initialized when the module is loaded) => when the prompt is instantiated a second time, the same object gets returned. Check out this loom for a debug preview: https://www.loom.com/share/2a500a4d83064489936886674e8641f9?sid=682f87fc-4052-4a33-89e4-5e4d1a6663dd

Solution: Changed the prompts to factory methods that returns a new prompt at every call

ssoima avatar Aug 10 '24 18:08 ssoima

@jjmachan , the failing lint check is not related to my code changes

ssoima avatar Aug 16 '24 10:08 ssoima

@jjmachan are you still following this?

ssoima avatar Sep 24 '24 19:09 ssoima

hey @ssoima with the new Prompt object this has been fixed

closing this for now but I'm really sorry we couldn't merge it 🙁 but at the same time thanks a million for taking the time to raise this, really grateful too and do checkout this form https://docs.google.com/forms/d/e/1FAIpQLSdM9FrrZrnpByG4XxuTbcAB-zn-Z7i_a7CsMkgBVOWQjRJckg/viewform - our way of saying thank you 🙂

jjmachan avatar Nov 03 '24 04:11 jjmachan