ableiweiss comments

Repositories
Issues
Comments

Results 3 comments of


                                            ableiweiss

Getting error "Failed to parse output. Returning None" on faithfulness metric

The issue is that `faithfulness_list` is empty, so it returns` np.nan`. The list `nli_result_text` looks fine, are there any unique requirements for the prompt that might cause this issue?

Getting error "Failed to parse output. Returning None" on faithfulness metric

I see that the last statement in `nli_result_text` get cut off and does not contain a `reason `or `verdict `field. The rest of the statements look good. Is there a...

Getting error "Failed to parse output. Returning None" on faithfulness metric

context window is 8k, tried LLama3-70b-instruct and Mixtral-8x-7b. Didn't have issues with the other metrics. Is there a particular open source LLM or context window you recommend to alleviate this?