autolabel icon indicating copy to clipboard operation
autolabel copied to clipboard

Provide a default `example_template` for different tasks when not specified by user

Open rishabh-bhargava opened this issue 2 years ago • 1 comments

Currently, if example_template is not provided, agent.plan fails with an unintuitive error:

In [6]: agent.plan('docs/assets/movie_reviews.csv')
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 agent.plan('docs/assets/movie_reviews.csv')

File ~/.pyenv/versions/3.8.13/envs/refuel-main/lib/python3.8/site-packages/autolabel/labeler.py:367, in LabelingAgent.plan(self, dataset, max_items, start_index)
    365 else:
    366     examples = []
--> 367 final_prompt = self.task.construct_prompt(input_i, examples)
    368 prompt_list.append(final_prompt)
    370 # Calculate the number of tokens

File ~/.pyenv/versions/3.8.13/envs/refuel-main/lib/python3.8/site-packages/autolabel/tasks/classification.py:51, in ClassificationTask.construct_prompt(self, input, examples)
     48     input[explanation_column] = ""
     50 # populate the current example in the prompt
---> 51 current_example = example_template.format_map(defaultdict(str, input))
     53 if self._is_few_shot_mode():
     54     return self.prompt_template.format(
     55         task_guidelines=fmt_task_guidelines,
     56         output_guidelines=self.output_guidelines,
     57         seed_examples="\n\n".join(fmt_examples),
     58         current_example=current_example,
     59     )

AttributeError: 'NoneType' object has no attribute 'format_map'

We could solve this in one of two ways:

  • provide a default example template for each task without a user having to provide example_template
  • throw an informative error for the user to provide example_template correctly

rishabh-bhargava avatar Jun 05 '23 05:06 rishabh-bhargava

Since this is a fairly straightforward part of the prompt and is not likely to change between different datasets/tasks (unlike the task_guidelines or obviously labels) I feel like it makes sense to have a default "Example: {example}\nOutput: {label}".

Tyrest avatar Jun 05 '23 23:06 Tyrest