unitxt icon indicating copy to clipboard operation
unitxt copied to clipboard

DiverseLabelsSamples clarification

Open yoavkatz opened this issue 1 year ago • 0 comments

DiverseLabelsSampler is used to select few shot examples.

  1. The name is not clear. It should be something DiverseDemosSampler.

  2. DiverseLabelsSampler requires adding a choices field - which requires adding a field to the task (although demo selection is not a property of the task:

  sampler=DiverseLabelsSampler(
        choices="choices", labels="is_faithful", include_empty_label=False
    ),

Suggested refinement implementation:

demos_sampler=DiverseDemosSampler(
        choices=["yes", "no"], label_field="is_faithful", include_empty_labels=False
    ),

or

    demos_sampler=DiverseDemosSampler(
        choices_field="choices", label_field="is_faithful", include_empty_labels=False
    )

if we want to keep also the old behavior.

yoavkatz avatar Mar 31 '24 14:03 yoavkatz