Help with .env.local for AWS as an endpoint for llama3 on huggingface cloud

[thams]

there seems to be no configuration for .env.local that I can get to work to connect to a Llama3 inference endpoint hosted by HuggingFace cloud (and I can find no examples).

MONGODB_URL=mongodb://localhost:27017
HF_TOKEN=hf_*******

MODELS=`[
  {
    "name": "AWS meta-llama-3-8b-pdf",
    "chatPromptTemplate": "<|system|>\n{{preprompt}}</s>\n{{#each messages}}{{#ifUser}}<|user|>\n{{content}}</s>\n<|assistant|>\n{{/ifUser}}{{#ifAssistant}}{{content}}</s>\n{{/ifAssistant}}{{/each}}",
    "parameters": {
      "temperature": 0.1,
      "top_p": 0.95,
      "repetition_penalty": 1.2,
      "top_k": 50,
      "truncate": 1000,
      "max_new_tokens": 2048,
      "stop": ["</s>"]
    },
    "endpoints": [
      {
        "url": "https://1212121212.us-east-1.aws.endpoints.huggingface.cloud"}
    ]
} 
]`

This flavor seems to need a value for endpoint type, one of:

Invalid discriminator value. Expected 'anthropic' | 'anthropic-vertex' | 'aws' | 'openai' | 'tgi' | 'llamacpp' | 'ollama' | 'vertex' | 'genai' | 'cloudflare' | 'cohere' | 'langserve'"

but none of them work.

[dlavrenuek]

@thams try it with

    "endpoints": [
      {
        "type": "tgi",
        "url": "https://1212121212.us-east-1.aws.endpoints.huggingface.cloud"
      }
    ]

Also not sure if your template works, at least llama does not seem to have the <|system|> token. This works for me:

    "chatPromptTemplate" : "<|begin_of_text|>{{#each messages}}{{#ifUser}}<|start_header_id|>user<|end_header_id|>\n{{#if @first}}{{#if @root.preprompt}}{{@root.preprompt}}\n{{/if}}{{/if}}{{content}}{{/ifUser}}{{#ifAssistant}}<|start_header_id|>assistant<|end_header_id|>\n{{content}}{{/ifAssistant}}{{/each}}",