unitxt
unitxt copied to clipboard
DiverseLabelsSamples clarification
DiverseLabelsSampler is used to select few shot examples.
-
The name is not clear. It should be something DiverseDemosSampler.
-
DiverseLabelsSampler requires adding a choices field - which requires adding a field to the task (although demo selection is not a property of the task:
sampler=DiverseLabelsSampler(
choices="choices", labels="is_faithful", include_empty_label=False
),
Suggested refinement implementation:
demos_sampler=DiverseDemosSampler(
choices=["yes", "no"], label_field="is_faithful", include_empty_labels=False
),
or
demos_sampler=DiverseDemosSampler(
choices_field="choices", label_field="is_faithful", include_empty_labels=False
)
if we want to keep also the old behavior.