dspy
dspy copied to clipboard
feature(dsp) Remove default max token param from OpenAI
For sending data to models with large context, this causes an artificial limit of outbound text and limits to max 8000 tokens. You are unable to set a large max_token value due to inherent limitations, and with a small value, it is hard to get more complex prompts to work for knowledge graph generation.
OpenAI sets it to infinite, which is optimal.
Hi @Vasilije1990 , thanks for the PR. unfortunately, this can't be merged since it will break existing caches. Why can you not set the max_tokens
within your LM configuration?