Trying to set up with TGI

[iChristGit]

I have installed TGI using docker, I can see the api docs at http://127.0.0.1:8080/docs/ But still cannot set up the env.local file, I have tried to set it up with the example, but always failing. Can someone who set it up correctly give me the rough idea of how to write the file ? I have tried a lot of combinations, and it always fail either internal error or the screenshot above.

[iChristGit]

Also tried:

MONGODB_URL = mongodb://127.0.0.1:27017
MODELS=`[
{
        "name": "HuggingFaceH4/zephyr-7b-beta",
        "datasetName": "HuggingFaceH4/ultrachat_200k",
        "description": "zephyr-7b-beta",
        "websiteUrl": "https://research.Trelis.com",
        "userMessageToken": "",
        "userMessageEndToken": " [/INST] ",
        "assistantMessageToken": "",
        "assistantMessageEndToken": " </s><s>[INST] ",
        "chatPromptTemplate" : "<s>[INST] <<SYS>>\nRespond in French to all questions\n<</SYS>>\n\n{{#each messages}}{{#ifUser}}{{content}} [/INST] {{/ifUser}}{{#ifAssistant}}{{content}} </s><s>[INST] {{/ifAssistant}}{{/each}}",
        "parameters": {
                "temperature": 0.01,
                "top_p": 0.95,
                "repetition_penalty": 1.2,
                "top_k": 50,
                "truncate": 1000,
                "max_new_tokens": 1024
        },
        "endpoints": [{
                "type" : "tgi",
                "url": "http://127.0.0.1:8080"
        }]
    }
]`   

[chittaranjan0275]

MONGODB_URL=mongodb://localhost:27017/ MODELS=[ { "name": "My LLM", "displayName": "mistralai/Mistral-7B-Instruct-v0.1", "description": "Mistral 7B is a new Apache 2.0 model, released by Mistral AI that outperforms Llama2 13B in benchmarks.", "websiteUrl": "https://mistral.ai/news/announcing-mistral-7b/", "preprompt": "", "chatPromptTemplate" : "<s>{{#each messages}}{{#ifUser}}[INST] {{#if @first}}{{#if @root.preprompt}}{{@root.preprompt}}\n{{/if}}{{/if}}{{content}} [/INST]{{/ifUser}}{{#ifAssistant}}{{content}}</s>{{/ifAssistant}}{{/each}}", "parameters": { "temperature": 0.1, "top_p": 0.95, "repetition_penalty": 1.2, "top_k": 50, "truncate": 3072, "max_new_tokens": 1024, "stop": ["</s>"] }, "promptExamples": [ { "title": "Write an email from bullet list", "prompt": "As a restaurant owner, write a professional email to the supplier to get these products every week: \n\n- Wine (x10)\n- Eggs (x24)\n- Bread (x12)" }, { "title": "Code a snake game", "prompt": "Code a basic snake game in python, give explanations for each step." }, { "title": "Assist in a task", "prompt": "How do I make a delicious lemon cheesecake?" } ] } ]

Try this, you can find this code in .env file. Don't forget to restart server.