unitxt
unitxt copied to clipboard
ensure standardized evaluation of CI across implementations
The new feature of Metric, sample-from-groups-scores, employs the CI over instances generated ad-hoc, one per group (a group is a subset of the input instances, whose member instances are those instances of the stream having a specific value in a specific field). CI generates the samples through first generating a list of indices (into the input instances) and then dealing with the instances thus indexed. Hence, the result of CI for these instances generated ad-hoc depends on the order in which these instances are fed into CI. This PR standardizes this order by the alphabetic order of the values that define the groups (as explained above)