openai_trtllm

openai_trtllm copied to clipboard

support for llama 3

4

will this project plan to support llama 3 70b or 8b?

avianion

llama 3 tokenizer no longer works - updated eos token

2

The official llama 3 70b instruct repo has updated the eos token "eos_token": "", Yet when using this library and using that eos token, no output is outputted because it...

avianion

ERROR: expected number of inputs between 1 and 3 but got 9 inputs for model

5

{"timestamp":"2024-04-15T05:20:55.796456Z","level":"ERROR","error":"AppError(error message received from triton: [request id: ] expected number of inputs between 1 and 3 but got 9 inputs for model 'myserving')","target":"openai_trtllm::routes::completions","span":{"headers":"{\"host\": \"localhost:3030\", \"user-agent\": \"OpenAI/Python 1.17.1\", \"content-length\": \"55\", \"accept\":...

samzong

[codellama] There is no space between each words

5

I deployed codellama on triton, but there is no space between each words on the stream mode ``` python openai_completion_stream.py beyondthecallofdutytosavehisfellowsoldiersafterthey�sufferedahorribleandfamiliarfateaftertheirmakingasuicidebarrageforanenemygunemplacement�(�Besalu,�GlobalWriteups). Dealing% ``` langchain_chat_openai works ``` python langchain_chat_openai.py Model regularization...

charllll