unitxt icon indicating copy to clipboard operation
unitxt copied to clipboard

ensure standardized evaluation of CI across implementations

Open dafnapension opened this issue 1 year ago • 0 comments

The new feature of Metric, sample-from-groups-scores, employs the CI over instances generated ad-hoc, one per group (a group is a subset of the input instances, whose member instances are those instances of the stream having a specific value in a specific field). CI generates the samples through first generating a list of indices (into the input instances) and then dealing with the instances thus indexed. Hence, the result of CI for these instances generated ad-hoc depends on the order in which these instances are fed into CI. This PR standardizes this order by the alphabetic order of the values that define the groups (as explained above)

dafnapension avatar May 17 '24 09:05 dafnapension