Johnno1011

Results 7 comments of Johnno1011

TGI crashes with complex json schemas provided as grammar without any information (on debug/trace level)

Hi! The LLM has to worry about generating the JSON as well as the fields in the schema, I think that's the issue. Grammar works 99% of the time really...

tensor_parallel method distributed=True

Ah I see what I was not doing right now, thanks for sharing this example and so quickly too! It looks like this is for distributed and tensor parallelism in...

OpenAI Client format + chat template for a single call

You could still use the openai.chat.completions.create but reset the chat history each time? For example: ``` def generate(prompt: str) -> ChatCompletion: messages = [ { "role": "system", "content": "You are...

[Feature] Support `response_format` for `TurboMind`

Hey, any updates on this issue?

TEI error for jinaai/jina-embeddings-v3 missing field `model_type` at line 51 column 1

I was able to resolve this by adding "model_type": "XLMRobertaModel" to the config.json of the downloaded model. Not a long term solution, but also not caused by TEI - the...

tgi server :: tool_choice="auto" behaves like tool_choice="required" from OpenAI spec

Hey guys, seen the PRs related to this that were in the recent release. It doesn't look like this has fixed the issue for me. I'm using HuggingFaceEndpoint wrapped up...

device-side assert triggered when trying to use LLaMA 3.2 Vision with grammar

Also experienced the same with this model, whether using grammar or just attempting to use the function_calling functionality