changing model to 30B in the .env file

[C0deXG]

here is the model am using which is 12B i want to change to 30B: defual one: MODELS=[ { "name": "OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5", "datasetName": "OpenAssistant/oasst1", "description": "A good alternative to ChatGPT", "websiteUrl": "https://open-assistant.io", "userMessageToken": "<|prompter|>", "assistantMessageToken": "<|assistant|>", "messageEndToken": "",`

this is what i change to: "name": "OpenAssistant/oasst-rlhf-2-llama-30b-7k-steps-xor", "datasetName": "OpenAssistant/oasst1", "description": "A good alternative to ChatGPT", "websiteUrl": "https://open-assistant.io", "userMessageToken": "<|prompter|>", "assistantMessageToken": "<|assistant|>", "messageEndToken": "</s>",

i got error when i run the model/chat-ui Model not found & Could not parse last message {"error":"Task not found for this model"} SyntaxError: Unexpected end of JSON input at JSON.parse (<anonymous>) at parseGeneratedText (/src/routes/conversation/[id]/+server.ts:178:32) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async saveMessage (/src/routes/conversation/[id]/+server.ts:94:26)

plz help if you know how to change the model to 30B OpenAssistant

[nsarrazin]

I think the reason you're having this issue is because there's no public endpoint available for this model ? See also #230

[C0deXG]

I think the reason you're having this issue is because there's no public endpoint available for this model ? See also #230

So how can I create my own endpoint font Open assistant 30B

[nsarrazin]

You can run it locally if you have the right hardware by running text-generation-inference.

Once you have it running, you can update your .env.local for chat-ui. In the MODELS variable add your new locally run model. You can see an example of a model config in the .env file

[aodrasa]

I'm having the same issue as OP and don't have the hardware to run it locally, do you know if there's a workaround? For context I'm trying to run the WizardLM-30B-Uncensored-GPTQ model.

[nsarrazin]

If you don't have the hardware to run it locally, your best bet would be to get a hosted solution somewhere, but there's not much we can do on this repo though.

Closing this issue but feel free to reopen if you feel I missed something!