Christian Otto Stelter

icon company PlusServer GmbH icon location Verl, Germany icon location Senior DevOps Engineer @plusserver, dog person. Currently building clouds based on @openstack and the tools to deploy and maintain these clouds.

Results 2 comments of Christian Otto Stelter

inference-workers exits when trying other models than distilgpt2 (on a non-GPU system)

No. I hoped for some feedback whether it is possible as in other solutions like fastchat (which does have an option for that). But after some testing with fastchat I...

When I'm using some third-party APIs, how can I add other supported models?

Take a look at https://github.com/imoneoi/openchat-ui/blob/main/types/openai.ts The OpenChat devs replaced it with their own models. They have their own OpenAI API compatible server.