Transformer Backend: Implementing use_tokenizer_template and stop_prompts options

[fakezeta]

Description

This PR implements embedded template in models for transformers backend with option:

template:
  use_tokenizer_template: true

in the yaml file.

This way the embedded chat template in the tokenizer_config.json is used.

stopwords:
- "<|end_of_text|>"
- "<|eot_id|>"

Option is also implemented. It accept a list of tokens that will be used as stopwords instead of the default.

Signed commits

• [X] Yes, I signed my commits.

[netlify[bot]]

✅ Deploy Preview for localai canceled.

Name	Link
🔨 Latest commit	43240311e69b26cde1596ef20d1446b4736e045f
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/662523ec7136fa00085f43cb