distilabel icon indicating copy to clipboard operation
distilabel copied to clipboard

[FEATURE] improved `Task` `prompt` customization

Open davidberenstein1957 opened this issue 5 months ago • 0 comments

Is your feature request related to a problem? Please describe.

  • where can I find the system prompt?
  • how easy is it to overwrite/adapt the prompt template?
  • can I leave out certain parts of the generation e.g. rationale/critique.

Describe the solution you'd like Some features/improvements that help with the questions above.

Describe alternatives you've considered NA

Additional context ideas which might not need to work with set_runtime_parameters specifically.

  • would it be possible to have a few-shot example section
Task.set_runtime_parameters(
    "few_shot_example": {
        "inputs": {"input_column": ["my example input"]}}, 
        "outputs": {"outut_column": ["my example output"]}}
    }
)
# or 
Task.set_runtime_parameters(
    "few_shot_example": "my good example"
)
  • can we allow adding a pre-fix/suf-fix to each prompt by default.
Task.set_runtime_parameters(
    "prompt_prefix": "always do this thingy",
    "prompt_suffix": "always do that other thingy"
)
  • always allow for adding a system prompt, even when this was not added in the original paper
Task.set_runtime_parameters(
    "system_prompt": "always do this thingy"
)
  • Overwrite prompt templates. should wee allow disabling certain output columns present in the prompt template through if-else statements? E.g., in UltraFeedback I might not want rationales.
Task.set_runtime_parameters(
    "prompt template": "do ultra feedback {score} raationale {rationale}"
)
Task.set_runtime_parameters(
    "prompt template": "do ultra feedback {score}"
)
Task.set_runtime_parameters(
    "prompt template": "path/to/.jinja2"
)

davidberenstein1957 avatar Sep 10 '24 12:09 davidberenstein1957