Add Outlines in LangChain

[rlouf]

We should open a PR on the LangChain repo to add Outlined as a model / guided generation provider.

[sandangel]

the tiltle is langchain but the description is llamaindex. I think they are different projects right? I’m interested in using Ollama in Langchain. If we add support for langchain that would be awesome.

[rlouf]

Yes it was supposed to be LangChain, thank you 😅

[pj-ml]

Is this happening soon?

[MightyGoldenJA]

If we could have an integration to langchain that would be really handy indeed.

[pj-ml]

@MightyGoldenJA I think you can use the outlines integration in vllm and pass it as an argument to the vllm integration in langchain (I hope I used the right phrasing). Something like this:

from langchain_community.llms import VLLM
from vllm.model_executor.guided_decoding.outlines_logits_processors import RegexLogitsProcessor
import vllm
from transformers import PreTrainedTokenizerFast

def placeholder_func(model_name, regex_string):
    llm = VLLM(
        model=model_name,
        trust_remote_code=True,
        # top_k=10,
        # top_p=0.95,
        temperature=0,
        vllm_kwargs={
            "sampling_params": vllm.SamplingParams(
                temperature=0.0,
                max_tokens=8_000,
                logits_processors=[
                    RegexLogitsProcessor(
                        regex_string=regex_string,
                        tokenizer=PreTrainedTokenizerFast.from_pretrained(model_name),
                    )
                ],
            ),
        },
    )
    return llm