unitxt
unitxt copied to clipboard
Provide access to raw predictions and references
Unitxt has two concepts:
(Raw/Texutal) Predictions and Processed Predictions (after applying post processing operators)
Today, the raw predictions which are added to the dataset during evaluation are transformed into the processed predictions, and there is no access to them in the metrics . Same for the textual references (e.g. can not run llm as judge on the raw textual predictions)
We should store the raw predications/raw predictions in MetricRecipe before they are transformed to by the Post processors and have them be available in the output dataset.
It would actually be best if the returned datasets would have these fields
"predictions"
"processed_predictions"
"references"
"processed_references"