unitxt icon indicating copy to clipboard operation
unitxt copied to clipboard

Provide access to raw predictions and references

Open yoavkatz opened this issue 1 year ago • 0 comments

Unitxt has two concepts:

(Raw/Texutal) Predictions and Processed Predictions (after applying post processing operators)

Today, the raw predictions which are added to the dataset during evaluation are transformed into the processed predictions, and there is no access to them in the metrics . Same for the textual references (e.g. can not run llm as judge on the raw textual predictions)

We should store the raw predications/raw predictions in MetricRecipe before they are transformed to by the Post processors and have them be available in the output dataset.

It would actually be best if the returned datasets would have these fields

"predictions"
"processed_predictions" "references" "processed_references"

yoavkatz avatar Jun 20 '24 06:06 yoavkatz