TPM issues when adding many new files

[rfuisz]

When given a lot of new "New file to index" files to work with, litellm seems to bounce of openai's TPM limits. It's possible that your TPM limits are much higher than mine, thus the issue:

| litellm.llms.OpenAI.openai.OpenAIError: Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-4o in organization org-NipyfbeUi3pFra7OYX5uNCJY on tokens per min (TPM): Limit 30000, Used 29554, Requested 812. Please try again in 732ms. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}} | | During handling of the above exception, another exception occurred: | | Traceback (most recent call last): | File "/Users/Main/Documents/Code/GitHub/outlawqa/submodules/paper-qa/paperqa/agents/search.py", line 374, in process_file | await tmp_docs.aadd( | File "/Users/Main/Documents/Code/GitHub/outlawqa/submodules/paper-qa/paperqa/docs.py", line 273, in aadd | result = await llm_model.run_prompt( | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Users/Main/Documents/Code/GitHub/outlawqa/submodules/paper-qa/paperqa/llms.py", line 181, in run_prompt | return await self._run_chat( | ^^^^^^^^^^^^^^^^^^^^^ | File "/Users/Main/Documents/Code/GitHub/outlawqa/submodules/paper-qa/paperqa/llms.py", line 234, in _run_chat | chunk = await self.achat(messages) | ^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Users/Main/Documents/Code/GitHub/outlawqa/submodules/paper-qa/paperqa/llms.py", line 440, in achat | response = await self.router.acompletion(self.name, messages) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/router.py", line 691, in acompletion | raise e | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/router.py", line 679, in acompletion | response = await self.async_function_with_fallbacks(**kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/router.py", line 2986, in async_function_with_fallbacks | raise original_exception | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/router.py", line 2846, in async_function_with_fallbacks | response = await self.async_function_with_retries(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/router.py", line 3037, in async_function_with_retries | self.should_retry_this_error( | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/router.py", line 3158, in should_retry_this_error | raise error | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/router.py", line 3023, in async_function_with_retries | response = await original_function(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/router.py", line 830, in _acompletion | raise e | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/router.py", line 802, in _acompletion | response = await _response | ^^^^^^^^^^^^^^^ | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/utils.py", line 1595, in wrapper_async | raise e | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/utils.py", line 1415, in wrapper_async | result = await original_function(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/main.py", line 443, in acompletion | raise exception_type( | ^^^^^^^^^^^^^^^ | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/utils.py", line 8189, in exception_type | raise e | File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/litellm/utils.py", line 6557, in exception_type | raise RateLimitError( | litellm.exceptions.RateLimitError: litellm.RateLimitError: RateLimitError: OpenAIException - Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-4o in organization org-NipyfbeUi3pFra7OYX5uNCJY on tokens per min (TPM): Limit 30000, Used 29554, Requested 812. Please try again in 732ms. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}} | Received Model Group=gpt-4o-2024-08-06 | Available Model Group Fallbacks=None +------------------------------------ Failed to execute tool call for tool gather_evidence. Traceback (most recent call last): File "/Users/Main/Documents/Code/GitHub/outlawqa/venv/lib/python3.12/site-packages/aviary/env.py", line 196, in _exec_tool_call content = await tool._tool_fn( ^^^^^^^^^^^^^^^^^^^^ File "/Users/Main/Documents/Code/GitHub/outlawqa/submodules/paper-qa/paperqa/agents/tools.py", line 221, in gather_evidence raise EmptyDocsError("Not gathering evidence due to having no papers.") paperqa.agents.tools.EmptyDocsError: Not gathering evidence due to having no papers.