microsoft
[Bug]: OpenAI API key changes not reflected across multiple calls
Do you need to file an issue?
- [x] I have searched the existing issues and this bug is not already filed.
- [x] My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here.
- [x] I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
Describe the bug
I am using graphrag in a FastAPI backend service where a request can create a new custom index. The OpenAI API keys appear to be cached in memory across requests. Even though configurations are set dynamically from a template each time (with a possible new key), and the correct key is present just before entering build_index function, the requests still use an outdated key. This appears in the following scenario: First, an incorrect api key is used, which results in an 401 error as expected. The next time a correct key is used, which still results in the same error showing an preview of the previous incorrect key. Logging the settings going into build_index shows the intended key.
My configuration explicitly sets caching to "none", but the documentation is unclear on whether this affects the overall caching behavior of the LLM client. Restarting the FastAPI server clears the issue, suggesting that the key is being held in memory rather than being refreshed dynamically. This might also be related to the use of the fnllm library as far as I understand.
Steps to reproduce
No response
Expected Behavior
No response
GraphRAG Config Used
# Paste your config here
Logs and screenshots
No response
Additional Information
- GraphRAG Version:
- Operating System:
- Python Version:
- Related Issues:
This is by design. It's not explicitly a caching issue (your key is not written to the cache), but is related to the cache and to threading. We use a singleton ModelFactory to register model instances so we can manage rate limiting with parallel API calls. The instance name is hard-coded per workflow step ("extract_graph" for example) to ensure the cache instance matches between runs. So this should be a first-one-wins situation, which is validated in your notes. I'll put it in the backlog to investigate a different approach, but we may not be able to get to it for awhile.
We haven't found a better way to manage this without disruptive architecture changes, so I'm going to close the issue. The primary issue is the registration of model instances by name within the ModelManager, which returns singletons. However, we now have a config option (on the v3/main branch, to be released in the next few weeks) for each workflow step to specify the model_instance_name. So, setting these to new values in the config object used in each request should return a new instance rather than the singleton.