Support model providers other than OpenAI and Azure

[natoverse]

Right now GraphRAG only natively supports models hosted by OpenAI and Azure. Many users would like to run additional models, including alternate APIs, SLMs, or models running locally. As a research team with limited bandwidth it is unlikely we will add native support for more model providers in the near future. Our focus is on memory structures and algorithms to improve LLM information retrieval, and we've got a lot of experiments in the queue!

There are alternative options to achieve extensibility, and many GraphRAG users have had luck extending the library. So far we've seen this most commonly with Ollama, which runs on localhost and supports a very wide variety of models. This approach depends on Ollama supporting the standard OpenAI API for chat completion and embeddings so it can proxy our API calls, and it looks like this is working for a lot of folks (though may require some hacking).

Please note: while we are excited to see GraphRAG used with more models, our team will not have time to help diagnose issues. We'll do our best to route bug reports to existing conversations that might be helpful. For the most part you should expect that if you file a bug related to running an alternate solution, we'll link to this issue, a relevant conversation if we're aware of one, and then we'll close the bug.

Here is a general discussion regarding OSS LLMs: https://github.com/microsoft/graphrag/discussions/321.

And a couple of popular Ollama-related issues: #339 and #345. We'll link to others in the comments when relevant.

Have a look at issues tagged with the community_support label as well.

[github-actions[bot]]

This issue has been marked stale due to inactivity after repo maintainer or community member responses that request more information or suggest a solution. It will be closed after five additional days.

[natoverse]

Here is a user-submitted solution for fixing the community rating value. We normally expect a float, but some models will send an int. https://github.com/microsoft/graphrag/issues/772

[nileshtrivedi]

If GraphRAG starts using Microsoft's own agent library AutoGen, supporting different model providers would not need to be re-implemented here.

LLM Configurations docs for AutoGen

[natoverse]

@Mxk-1 has found chunking settings that help resolve issues with create_base_entity_graph with Ollama:

The chunk splitting in the original setting.yaml provided may not be suitable for the model launched with Ollama, as it could be either too large or too small, leading to errors in the model's responses. The original paper mentioned using the GPT-4o model, while the model I deployed locally is Gemma2:9b via Ollama. These two models differ in size and performance.

Additionally, since the pipeline relies on prompt-based Q&A with the text, the prompt itself takes up some of the model's processing length. By adjusting the chunk_size, I was able to successfully run the experiment. If you encounter this issue, try increasing or decreasing the chunk_size. If you have a better solution, feel free to discuss it with me.

[MortalHappiness]

I collected several comments scattered across different issues and created a monkey patch script along with a working setting for Ollama. It has been tested on version 0.3.2 and works properly. I’m sharing it for those who might need it: https://gist.github.com/MortalHappiness/7030bbe96c4bece8a07ea9057ba18b86.

I’m not sure if it’s appropriate to comment here, so if the reviewers think it’s not, I'll delete this comment and post it in a more suitable place. Thank you in advance!

[satyaloka93]

Please consider just using allowing openai compatible endpoint (vllm/llama-server) for llm and embedding model. I can get it to work for the normal llm, but not to make embeddings (nomic-embed-text) via llama-server. Please don't shoehorn this into only ollama, yet creating another niche constriction for usability.

@MortalHappiness does your monkey-patch work with other openai locally hosted endpoints that are not ollama?

[ishaan-jaff]

Hi @natoverse @MortalHappiness @satyaloka93 made a PR to add litellm - support for Ollama, Vertex AI, Gemini, Anthropic, Bedrock (100+LLMs)

https://github.com/microsoft/graphrag/pull/1575

adds support for the above mentioned LLMs using LiteLLM https://github.com/BerriAI/litellm/ LiteLLM is a lightweight package to simplify LLM API calls - use any llm as a drop in replacement for gpt-4o.

Example

from litellm import completion
import os

## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-cohere-key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="openai/gpt-4o", messages=messages)

# anthropic call
response = completion(model="anthropic/claude-3-sonnet-20240229", messages=messages)
print(response)

Response (OpenAI Format)

{
    "id": "chatcmpl-565d891b-a42e-4c39-8d14-82a1f5208885",
    "created": 1734366691,
    "model": "claude-3-sonnet-20240229",
    "object": "chat.completion",
    "system_fingerprint": null,
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "content": "Hello! As an AI language model, I don't have feelings, but I'm operating properly and ready to assist you with any questions or tasks you may have. How can I help you today?",
                "role": "assistant",
                "tool_calls": null,
                "function_call": null
            }
        }
    ],
    "usage": {
        "completion_tokens": 43,
        "prompt_tokens": 13,
        "total_tokens": 56,
        "completion_tokens_details": null,
        "prompt_tokens_details": {
            "audio_tokens": null,
            "cached_tokens": 0
        },
        "cache_creation_input_tokens": 0,
        "cache_read_input_tokens": 0
    }
}

[natoverse]

Note that as of GraphRAG 2.0 we allow injection of custom model providers. There are some details on how to do this in the new models docs page.

[aliok]

Note that as of GraphRAG 2.0 we allow injection of custom model providers. There are some details on how to do this in the new models docs page.

I gave this a try with various model providers.

• Google Gemini: doesn't support frequency_penalty, even though they have an OpenAI compatible API. Tried with gemini-2.5-pro as the chat model and used a non-Gemini model outside of Google for the embedding model.
• AWS Bedrock: Tried with Claude Sonnet 3.5 as the chat model and Amazon Titan text embedding as the embedding model. Got an error that only floats are accepted as encoding format.
• Local models via Ollama: looking for a hint here...

I saw some tutorials online that people made GraphRAG working with local models, but mentioned models are very old (Llama 3.1, etc). Can anybody in GraphRAG team, or in the community suggest a well working pair of chat and embedding models that are in https://ollama.com/search ?

[nhha1602]

I tried to use vLLM to load Qwen 3 14B AWQ model. But I get following error: KeyError: 'community' Could you, please advise this and how to fixed this ?

2025-08-20 16:47:25.0124 - ERROR - graphrag.index.operations.summarize_communities.community_reports_extractor - error generating community report Traceback (most recent call last): File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/graphrag/index/operations/summarize_communities/community_reports_extractor.py", line 80, in call response = await self._model.achat( File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/graphrag/language_model/providers/fnllm/models.py", line 86, in achat output=BaseModelOutput( File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/pydantic/main.py", line 253, in init validated_self = self.pydantic_validator.validate_python(data, self_instance=self) pydantic_core._pydantic_core.ValidationError: 1 validation error for BaseModelOutput content Input should be a valid string [type=string_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.11/v/string_type 2025-08-20 16:47:25.0125 - ERROR - graphrag.index.operations.summarize_communities.strategies - Community Report Extraction Error Traceback (most recent call last): File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/graphrag/index/operations/summarize_communities/community_reports_extractor.py", line 80, in call response = await self._model.achat( File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/graphrag/language_model/providers/fnllm/models.py", line 86, in achat output=BaseModelOutput( File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/pydantic/main.py", line 253, in init validated_self = self.pydantic_validator.validate_python(data, self_instance=self) pydantic_core._pydantic_core.ValidationError: 1 validation error for BaseModelOutput content Input should be a valid string [type=string_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.11/v/string_type 2025-08-20 16:47:25.0125 - WARNING - graphrag.index.operations.summarize_communities.strategies - No report found for community: 0.0 2025-08-20 16:47:25.0125 - INFO - graphrag.logger.progress - level 0 summarize communities progress: 1/1 2025-08-20 16:47:25.0127 - ERROR - graphrag.index.run.run_pipeline - error running workflow create_community_reports Traceback (most recent call last): File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/graphrag/index/run/run_pipeline.py", line 108, in _run_pipeline result = await workflow_function(config, context) File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/graphrag/index/workflows/create_community_reports.py", line 64, in run_workflow output = await create_community_reports( File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/graphrag/index/workflows/create_community_reports.py", line 130, in create_community_reports return finalize_community_reports(community_reports, communities) File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/graphrag/index/operations/finalize_community_reports.py", line 19, in finalize_community_reports community_reports = reports.merge( File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/pandas/core/frame.py", line 10839, in merge return merge( File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/pandas/core/reshape/merge.py", line 170, in merge op = _MergeOperation( File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/pandas/core/reshape/merge.py", line 794, in init ) = self._get_merge_keys() File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/pandas/core/reshape/merge.py", line 1311, in _get_merge_keys left_keys.append(left._get_label_or_level_values(lk)) File "/home/abc/anaconda3/envs/graphrag/lib/python3.10/site-packages/pandas/core/generic.py", line 1911, in _get_label_or_level_values raise KeyError(key) KeyError: 'community' 2025-08-20 16:47:25.0133 - ERROR - graphrag.api.index - Workflow create_community_reports completed with errors 2025-08-20 16:47:25.0134 - ERROR - graphrag.cli.index - Errors occurred during the pipeline run, see logs for more details.

[laszukdawid]

I just used gpt-5-mini with embeddings text-embedding-3-small and got nothing in return, i.e. KeyError: "Column(s) ['description', 'source_id', 'weight'] do not exist". The output is empty; stats.json contains only 0s and context.json is {}.

I wouldn't necessarily opt to support all models from all providers, as they can work differently with different prompts. However, I'd urge to have some integration test for models from the same provider.

Alternatively, a list of provider/model which has been verified on different document corpora.

[natoverse]

This is coming soon - OpenAI changed the format of their response, so we had to update our LLM management library. Should be releasing with 2.6.0 in the next couple of days.

[natoverse]

2.6.0 is available now. Not only does it resolve the format issues from OpenAI, it also introduces the option to use LiteLLM to configure any model provider that they support.

[gona-sreelatha]

If it’s helpful to others, I don’t think this issue is related to the model itself.

I’ve seen similar errors such as:

1. ValueError: Columns must be same length as key
2. KeyError: "Column(s) ['description', 'source_id', 'weight'] do not exist"
3. KeyError: 'title' (from pandas\core\groupby\grouper.py)

All of these occur when the files being indexed don’t contain enough meaningful text for GraphRAG to extract any entities or relationships. In such cases, the extraction step returns empty DataFrames, which then cause downstream failures during merging or grouping.

It would be great if GraphRAG could handle this case more gracefully — for example, by skipping empty files or checking for malfomred dfs returned from llm or atleast throw better exception than breaking down at pandas as mentioned above