Gideon Giffard

Prompt templating is available in [v4.34](https://github.com/huggingface/transformers/releases/tag/v4.34.0): > We've added a new [template](https://huggingface.co/docs/transformers/main/chat_templating) feature for chat models. This allows the formatting that a chat model was trained with to be saved...

Thanks @Tushar-ml !

We've added a [temporary workaround](https://github.com/hyperonym/basaran/pull/199/files) in [v0.18.1](https://github.com/hyperonym/basaran/releases/tag/v0.18.1): currently only the first prompt in the list will be used, and `400 Bad Request` will be returned if the prompt list contains...

Default template added in https://github.com/hyperonym/basaran/pull/163

It appears that the error originates from the internal code of MPT. We will conduct further testing.

We've ran into the exact same error before: https://github.com/hyperonym/basaran/issues/5. The error is caused by https://github.com/TimDettmers/bitsandbytes/issues/162 and seems fully random. Currently the only workaround is to stop using `INT8` quantization, and...

I think there might be two possible reasons for the `404` error: 1. Please make sure `api_base` is set correctly. Specifically: | api_base | | | :--- | --- |...

Thanks for introducing TGU, we'll definitely check it out!