Conversation templates Api

[aolivtous]

Hi! I would like to know if the conversation templates can be changed. I am using two models T5 and vicuna-7b. The default T5 conversation template is long and reduces the input size of the context (2048) in the tokenization. So, with T5 I managed to remove the conversation template in the huggingface_api.py just passing the context + question as a message. Like this:

def main(args, msg, model, tokenizer): #conv = get_conversation_template(args.model_path) #conv.append_message(conv.roles[0], msg) #conv.append_message(conv.roles[1], None) #prompt = conv.get_prompt() # input_ids = tokenizer([prompt]).input_ids

input_ids = tokenizer([msg]).input_ids

output_ids = model.generate(
    torch.as_tensor(input_ids).cuda(),
    do_sample=True,
    temperature=args.temperature,
    max_new_tokens=args.max_new_tokens,
)
if model.config.is_encoder_decoder:
    output_ids = output_ids[0]
else:
    output_ids = output_ids[0][len(input_ids[0]) :]
outputs = tokenizer.decode(
    output_ids, skip_special_tokens=True, spaces_between_special_tokens=False
) 

return outputs

However, If I do the same but using the weights from vicuna-7B the outputs are always "." . I tried to use the conversation template for Vicuna and it works. I also tried to put the system message = "" and it also works. So, why is the conversation template necessary for vicuna 7-B but not for T5? What happens if I change the conversation template for inference and fine-tuning?