Different format for inference ？

[PansaLegrand]

[PansaLegrand]

When training, the model sees the input in a format like: "Below is an instruction that describes a task, paired with an input that provides further context. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:"

For inference, do we add the first line (or something else) to the user input before sending into the model? Or only the user input ?

[sergevar]

What we've learned by running a character-style chat inference with Alpaca is that for the majority of use cases, the strictness of adherence to this format is not required, the model is flexible enough to adapt to all kinds of input. But it might affect accuracy slightly, how much is anyone's guess right now.