text-generation-inference
text-generation-inference copied to clipboard
huggingface
Guidance acceleration
Feature request
Guidance can control the generated format, which could be a nice feature if it is built-in
- Add extra parameter to
/generateand/generate_streamprotocol to specify a template - Use guidance-like mask to generate tokens and pre-fill known tokens, alternately
Motivation
See https://github.com/microsoft/guidance#guidance-acceleration-notebook
Your contribution
Just an idea for now
For now, won't it be better to add text-generation-inference as a supported LLM into Guidance, in the same way as it did for OpenAI's models?
Right, that's possible, but built-in support would "significantly improve inference performance".
When multiple generation or LLM-directed control flow statements are used in a single Guidance program then we can significantly improve inference performance by optimally reusing the Key/Value caches as we progress through the prompt. This means Guidance only asks the LLM to generate the green text below, not the entire program. This cuts this prompt's runtime in half vs. a standard generation approach.
Similar feature in another text generation project: https://github.com/ggerganov/llama.cpp/pull/1773
Similar feature in another text generation project: ggerganov/llama.cpp#1773
I wonder konw if it can or not get a stream with batch process
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}