autolabel
autolabel copied to clipboard
Provide a default `example_template` for different tasks when not specified by user
Currently, if example_template is not provided, agent.plan fails with an unintuitive error:
In [6]: agent.plan('docs/assets/movie_reviews.csv')
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 agent.plan('docs/assets/movie_reviews.csv')
File ~/.pyenv/versions/3.8.13/envs/refuel-main/lib/python3.8/site-packages/autolabel/labeler.py:367, in LabelingAgent.plan(self, dataset, max_items, start_index)
365 else:
366 examples = []
--> 367 final_prompt = self.task.construct_prompt(input_i, examples)
368 prompt_list.append(final_prompt)
370 # Calculate the number of tokens
File ~/.pyenv/versions/3.8.13/envs/refuel-main/lib/python3.8/site-packages/autolabel/tasks/classification.py:51, in ClassificationTask.construct_prompt(self, input, examples)
48 input[explanation_column] = ""
50 # populate the current example in the prompt
---> 51 current_example = example_template.format_map(defaultdict(str, input))
53 if self._is_few_shot_mode():
54 return self.prompt_template.format(
55 task_guidelines=fmt_task_guidelines,
56 output_guidelines=self.output_guidelines,
57 seed_examples="\n\n".join(fmt_examples),
58 current_example=current_example,
59 )
AttributeError: 'NoneType' object has no attribute 'format_map'
We could solve this in one of two ways:
- provide a default example template for each task without a user having to provide
example_template - throw an informative error for the user to provide
example_templatecorrectly
Since this is a fairly straightforward part of the prompt and is not likely to change between different datasets/tasks (unlike the task_guidelines or obviously labels) I feel like it makes sense to have a default "Example: {example}\nOutput: {label}".