stepoibm comments

Repositories
Issues
Comments

Results 3 comments of


                                            stepoibm

Invalid response format. Expected a list of dictionaries with keys 'verdict'

I've got the same issue with mixtral 8x7B. Sometimes the llm will generate `classification: [{"statement_1": "England won the 2022 ICC Men's T20 World Cup.", "reason": "From context it is clear...

Invalid response format. Expected a list of dictionaries with keys 'verdict'

@ajaychinni I've "fixed" the issues on a fork for myself: https://github.com/explodinggradients/ragas/pull/633 This should fix your "Attributed" and "verdict" issues

Invalid response format. Expected a list of dictionaries with keys 'verdict'

We are using mixtral. The issue is that emitting the expected json perfectly is super fragile for other models. That's not a RAGAS issue, but a model issue. LLama didnt...