Using LangGraph with Models on HuggingFace

[PouriaRouzrokh]

Hi!

Can someone use this package with open-source models, like those released on HF, instead of the OpenAI models?

Thanks!

[hinthornw]

A node can be any python function - so absolutely! Any particular example you'd like us to highlight?

[seyi33]

A node can be any python function - so absolutely! Any particular example you'd like us to highlight?

I have tried this but quite unsuccessful for now, can you try giving an example with this "repo_id="mistralai/Mixtral-8x7B-Instruct-v0.1""

[AnouarZerrik]

I use Gemini Pro, but I have an error:

KeyError Traceback (most recent call last) in <cell line: 1>() ----> 1 for s in graph.stream( 2 { 3 "messages": [ 4 HumanMessage( 5 content="Fetch the UK's GDP over the past 5 years,"

28 frames /usr/local/lib/python3.10/dist-packages/langchain/agents/react/agent.py in (x) 104 agent = ( 105 RunnablePassthrough.assign( --> 106 agent_scratchpad=lambda x: format_log_to_str(x["intermediate_steps"]), 107 ) 108 | prompt

KeyError: 'intermediate_steps'

[PlebeiusGaragicus]

This is my intent as well - will report back with my progress. Currently had success running with Ollama on an M2 macbook.

[PlebeiusGaragicus]

1. I used the provided example (https://github.com/langchain-ai/langgraph/blob/main/examples/agent_executor/base.ipynb)
2. I imported the Ollama LLM generator from langchain_community.llms.ollama import Ollama
3. I changed the code from llm = ChatOpenAI(model="gpt-3.5-turbo-1106", streaming=True) to llm = Ollama(model='llama2')
4. I get a runtime error:

  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 1497, in _transform_stream_with_config
    chunk: Output = context.run(next, iterator)  # type: ignore
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langgraph/pregel/__init__.py", line 347, in _transform
    _interrupt_or_proceed(done, inflight, step)
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langgraph/pregel/__init__.py", line 688, in _interrupt_or_proceed
    raise exc
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 4041, in invoke
    return self.bound.invoke(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 2053, in invoke
    input = step.invoke(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 3507, in invoke
    return self._call_with_config(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 1246, in _call_with_config
    context.run(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/config.py", line 326, in call_func_with_variable_args
    return func(input, **kwargs)  # type: ignore[call-arg]
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 3383, in _invoke
    output = call_func_with_variable_args(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/config.py", line 326, in call_func_with_variable_args
    return func(input, **kwargs)  # type: ignore[call-arg]
  File "/Users/plebbyg/Downloads/agents_test/agents_test/flows/agent_exec.py", line 127, in run_agent
    agent_outcome = agent_runnable.invoke(data)
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 2053, in invoke
    input = step.invoke(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 4041, in invoke
    return self.bound.invoke(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 235, in invoke
    self.generate_prompt(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 530, in generate_prompt
    return self.generate(prompt_strings, stop=stop, callbacks=callbacks, **kwargs)
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 703, in generate
    output = self._generate_helper(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 567, in _generate_helper
    raise e
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/llms.py", line 554, in _generate_helper
    self._generate(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_community/llms/ollama.py", line 408, in _generate
    final_chunk = super()._stream_with_aggregation(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_community/llms/ollama.py", line 317, in _stream_with_aggregation
    for stream_resp in self._create_generate_stream(prompt, stop, **kwargs):
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_community/llms/ollama.py", line 159, in _create_generate_stream
    yield from self._create_stream(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_community/llms/ollama.py", line 240, in _create_stream
    raise ValueError(
ValueError: Ollama call failed with status code 400. Details: invalid options: functions

I have two questions: (1) I'm unsure which models have been fine-tuned for OpenAI-style tool usage and (2) I wonder if the Ollama package needs to be updated for this usage, regardless.

[PlebeiusGaragicus]

After digging deeper I found a wrapper for Ollama that should add function calling capability: from langchain_experimental.llms.ollama_functions import OllamaFunctions

Now, setting up my llm as llm = OllamaFunctions() I get a KeyError:

  File "/Users/plebbyg/Downloads/agents_test/agents_test/flows/agent_exec.py", line 138, in run_agent
    agent_outcome = agent_runnable.invoke(data)
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 2053, in invoke
    input = step.invoke(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 4041, in invoke
    return self.bound.invoke(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 166, in invoke
    self.generate_prompt(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 544, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 408, in generate
    raise e
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 398, in generate
    self._generate_with_cache(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 577, in _generate_with_cache
    return self._generate(
  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_experimental/llms/ollama_functions.py", line 103, in _generate
    called_tool_name = parsed_chat_result["tool"]
KeyError: 'tool'

[jchaykow]

After digging deeper I found a wrapper for Ollama that should add function calling capability: from langchain_experimental.llms.ollama_functions import OllamaFunctions

Now, setting up my llm as llm = OllamaFunctions() I get a KeyError:

  File "/Users/plebbyg/Downloads/agents_test/agents_test/flows/agent_exec.py", line 138, in run_agent

    agent_outcome = agent_runnable.invoke(data)

  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 2053, in invoke

    input = step.invoke(

  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 4041, in invoke

    return self.bound.invoke(

  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 166, in invoke

    self.generate_prompt(

  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 544, in generate_prompt

    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)

  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 408, in generate

    raise e

  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 398, in generate

    self._generate_with_cache(

  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 577, in _generate_with_cache

    return self._generate(

  File "/Users/plebbyg/Downloads/agents_test/venv/lib/python3.9/site-packages/langchain_experimental/llms/ollama_functions.py", line 103, in _generate

    called_tool_name = parsed_chat_result["tool"]

KeyError: 'tool'

Getting same key error. Also seems to be a new error as it was working a few days ago.

[dtubb]

Any updates on running LangGraph locally using Ollama? I'm got a LangGraph work quite well on an archive. But, it quickly got expensive running on ChatGPT. Would love to be bale to keep playing, without the marginal cost. Are there examples of get a LangGraph going, using Ollama, and say one of the Msitral models?

[Nov1ceLee]

Got Any Langgraph examples with not-supported function call model? in the notebooks, the examples are all ChatGPT

[dtubb]

I've not had a chance to play, but this is a demo, using Ollama, on Mistral. Seems like it was just posted.

examples/rag/langgraph_crag_mistral.ipynb

[DonikHronic]

Got Any Langgraph supervisor examples without openai function? For example, can I create the supervisor using only prompt or langchain tool instead of function definition?

[hinthornw]

@DonikHronic you don't need function calling for multi-agent to work. Especially if you use things like single-input tools, you could just send the strings directly to many of the nodes.

The reason I use function/tool calling in my examples is it typically creates fewer headaches for most developers (you have to worry a lot less about parsing/reparsing/validating/etc), and any time I put out a notebook designed for LLM's that aren't adapted for function calling, i get a slew of complaints about it's not up to date with industry standards or not prod ready.

[hinthornw]

We are trying to address this by adding more support for with_structured_output and bind_tools support to make it easier to generate and parse .

But it's labor intensive to add and maintain for support across all integrations, so support for it is somewhat depends on how active the community and partner is around each integration

[hinthornw]

The LLMComplier example does use a custom output format, and I tested it on Mixtral previously with some good success. You could try that one as well

[dongfangduoshou123]

@hinthornw From the user's perspective, I believe that both frameworks and examples should prioritize universality rather than extensive efforts to adapt to open-ai. In actual production, considering data security, the majority of models are likely to be inclined towards calling a privately deployed model. Therefore, methods such as bind_tools and optimizations with_structured_output are crucial, and it's also important to provide examples based on bind_tools and with_structured_output. Thank you!

[hinthornw]

I do agreed that supporting structured outputs / reliable function calling across as many modes as possible is a priority. Most users/enterprises don't share all the same priorities as the ones you've listed though so it's still a work in progress.

[StuartJAtkinson]

I see this is getting traction the last few examples seem to focus hugging face which is great. Frankly I don't know when to mark this complete I'm at a point where I think I've seen enough examples put out I just don't have time to catch up with haha. I suppose rather than the examples because obviously they're quite use case based and there's many paths to forming solutions, the double edged sword of open source. I don't know if I should close this and raise a new one that focuses on "Anatomy of Langchain" documentation wise. I'm looking at making a vector based mind-map/informatic for the 'categories' because people in the know know the distinction between "tasks" data types, functions (like inference, quantization etc. but for people wanting to start the distinction between use cases and features are difficult to even distinguish since the examples just go "We're going to use this <Cool OpenSource Cute name like LLama, StrawberrySunset whatever> feature" haha. So while I appreciate this connection and some decent examples have come out I have no context as to where they exist in the AI stack.

[lovebeatz]

A node can be any python function - so absolutely! Any particular example you'd like us to highlight?

so basically, I can use any inferencing library out there in the world while including memory features like MEMGPT without ever having to import any Langchain module, am I correct?

[PlebeiusGaragicus]

Not sure what you mean by "having to import any Langchain module." You're talking about LangGraph - right? So you'd have to import that much, but it doesn't force you to use the Langchain LLM wrappers inside the node.

I think the answer is yes - you can create a LangGraph node which is just arbitrary python code. As long as you're respecting the larger LangGraph structure as far as the node code goes - that is, taking the graph state and config as parameters and returning an update to the graph state - my understanding is that you'd be able to run anything you want inside that node.

The only problem I've been having is that LangGraph will start threads as it computes the graph so you need to ensure everything is thread-safe.

[lovebeatz]

Not sure what you mean by "having to import any Langchain module." You're talking about LangGraph - right? So you'd have to import that much, but it doesn't force you to use the Langchain LLM wrappers inside the node.

I think the answer is yes - you can create a LangGraph node which is just arbitrary python code. As long as you're respecting the larger LangGraph structure as far as the node code goes - that is, taking the graph state and config as parameters and returning an update to the graph state - my understanding is that you'd be able to run anything you want inside that node.

The only problem I've been having is that LangGraph will start threads as it computes the graph so you need to ensure everything is thread-safe.

Yeah call it a wrapper! Okay, a few follow-up questions here:

1. LanGgraph is production ready right? I can deploy and serve it to 10k people right?
2. Let's not go around talking broadly about what I wanna run inside node- let me state it here- I want to run exllamav2 kernels for fast inference of AWQ quantized models, currently it is supported by AutoAWQ and huggingface transformers, my question is shall I fun codes of these libraries in the node or rather try to import langchain to call support wrappers, although not a direct integration of AutoAWQ in langchain, I might have to go with Aphrodite API or go via vllm route
3. What about the threads? Would there be any difference in running langchain wrappers inside nodes or third party ones
4. What exactly gives langchain wrappers advantage here, as far as node code is concerned?

Baseline is I would be implementing memGPT, Saving conversation of every user, and all of it will be deployed on kubernetes running docker with load balanced among tens of GPU VMs

Thanks lol

[lovebeatz]

Posting another set of questions: Given my use-case I would only be using either groqcloud api for testing purposes of self hosted LLM for deployment purposes

My question(s) given that the tutorials here and there mention function calling, are as follows:

1. How much would the functionality loss be if turning my head away from the assistants API
2. Given that I am not considering fine-tuning for function calling, can the behaviour be simulated by writing custom python function additionally to give the LLM the function calling essence
3. How to determine at a glance, if there are specific identifiers to know that given implementation features function calling (like examples in the repo or some YouTube tutorial etc)