graphiti icon indicating copy to clipboard operation
graphiti copied to clipboard

[BUG] LLMConfig.max_tokens not respected by OpenAI-based clients

Open paveljakov opened this issue 5 months ago • 6 comments

LLMConfig.max_tokens not respected by OpenAI-based clients

Bug Description

When configuring max_tokens in LLMConfig, the setting is ignored by OpenAI-based clients (OpenAIClient and AzureOpenAILLMClient), causing token limit errors even when a higher limit is configured.

Additionally, there are currently two confusing ways to set max_tokens for OpenAI clients:

  1. Through LLMConfig(max_tokens=...)
  2. Through the client constructor parameter OpenAIClient(max_tokens=...)

This dual configuration is confusing because:

  • It's unclear which one takes precedence
  • Users naturally expect the config object to be the authoritative source
  • The constructor parameter silently overrides the config setting
  • Other clients (Gemini, Anthropic) don't have this confusing dual interface

What I was trying to do

Configure a higher token limit for LLM responses to avoid truncation:

from graphiti_core.llm_client.config import LLMConfig
from graphiti_core.llm_client.openai_client import OpenAIClient

config = LLMConfig(
    api_key="...",
    model="gpt-4",
    max_tokens=16000  # Set higher limit
)

client = OpenAIClient(config=config)

What I expected to happen

The client should respect the max_tokens=16000 setting and allow responses up to 16,000 tokens, similar to how other configuration options work.

What actually happened

The client still uses the default 8192 token limit, resulting in errors like: Output length exceeded max tokens 8192: Could not parse response content as the length limit was reached

Root Cause

In BaseOpenAIClient.__init__(), the constructor parameter max_tokens=DEFAULT_MAX_TOKENS always overrides config.max_tokens, even when explicitly set by the user:

# Current problematic code:
def __init__(self, config: LLMConfig | None = None, cache: bool = False, max_tokens: int = DEFAULT_MAX_TOKENS):
    # ...
    super().__init__(config, cache)
    self.max_tokens = max_tokens  # This overwrites config.max_tokens!

Code sample that demonstrates the issue

from graphiti_core.llm_client.config import LLMConfig
from graphiti_core.llm_client.openai_client import OpenAIClient

config = LLMConfig(max_tokens=16000)
client = OpenAIClient(config=config)

print(f"Config max_tokens: {config.max_tokens}")  # Shows: 16000
print(f"Client max_tokens: {client.max_tokens}")  # Shows: 8192 (wrong!)

# User expects: client.max_tokens == 16000
# Actual result: client.max_tokens == 8192

Affected Clients

  • OpenAIClient
  • AzureOpenAILLMClient

Other clients handle this correctly:

  • GeminiClient - respects config and has clear precedence rules
  • AnthropicClient - respects config settings
  • GroqClient - respects config settings

Proposed Solution

The fix should:

  1. Respect config.max_tokens when explicitly set (different from default)
  2. Use constructor parameter as fallback when config has default value
  3. Maintain backward compatibility for existing code
  4. Clear precedence rules: Config takes priority over constructor parameter
# Proposed fix:
if config.max_tokens != DEFAULT_MAX_TOKENS:
    self.max_tokens = config.max_tokens  # Use config when explicitly set
else:
    self.max_tokens = max_tokens  # Use constructor parameter as fallback

Environment

  • Graphiti version: [current version]
  • Python version: 3.12+
  • Affected clients: OpenAIClient, AzureOpenAILLMClient

Labels

  • bug
  • help wanted
  • good first issue
  • llm-client

paveljakov avatar Jul 24 '25 15:07 paveljakov

Thanks for the PR. I've left some comments.

danielchalef avatar Jul 27 '25 01:07 danielchalef

same! I still have this issue on the latest version. causing issues.

修改代码后仍然报相同的错

soberyyz avatar Jul 31 '25 07:07 soberyyz

Hi - Any update on the fix for this getting merged in?

tha23rd avatar Aug 21 '25 13:08 tha23rd

@paveljakov Is this still an issue? Please confirm within 14 days or this issue will be closed.

claude[bot] avatar Oct 06 '25 00:10 claude[bot]

@paveljakov Is this still an issue? Please confirm within 14 days or this issue will be closed.

claude[bot] avatar Nov 17 '25 00:11 claude[bot]