[Bug]: logs blowing up with `Cannot add callback - would exceed MAX_CALLBACKS limit of 30.`

[jamesbraza]

What happened?

Running an evaluation script made a few months ago, with modern litellm my logs get immediately demolished with many lines saying:

LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30

I don't know what this is, the script does not configure 30 callbacks. This seems to be a new behavior added sometime in the past 3 months (https://github.com/BerriAI/litellm/pull/8112).

Some possible resolutions include:

1. Adding context to this message: what is the callback related to, why this error matters, possible resolutions
2. Removing this warning, or designing control flows to not have this edge case

In the meantime, this goes away as of litellm==1.58.4 (pip install "litellm<1.59").

Relevant log output

2025-04-06 17:21:37,193 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,193 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,194 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,194 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,194 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,194 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,194 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,194 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,194 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,194 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,195 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,195 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,195 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,195 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,195 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,196 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,196 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,196 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,196 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,196 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,196 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,196 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,197 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,197 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,197 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,197 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,197 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,197 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,198 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,198 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,198 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,198 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,198 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,198 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30
2025-04-06 17:21:37,199 - LiteLLM - WARNING - Cannot add callback - would exceed MAX_CALLBACKS limit of 30. Current callbacks: 30

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

1.65.4.post1

Twitter / LinkedIn details

No response

[daaain]

I'm also getting this after updating the version, not sure why?

I'm only setting 2 callbacks on the top level:

litellm.success_callback = ["langfuse"]
litellm.failure_callback = ["langfuse"] 

Another thing I changed is to create a new LiteLLM Router instance before each call, because I kept getting litellm.APIConnectionError: <asyncio.locks.Event object at 0x3ed4201d28a0 [unset]> is bound to a different event loop but that shouldn't add new listeners?

Would be great to have a way to debug this by listing the listeners or something?

[jamesbraza]

With litellm==1.67.4.post1, I disabled this message via:

import litellm


def update_litellm_max_callbacks(value: int = 1000) -> None:
    """Update litellm's MAX_CALLBACKS limit, can call with default to defeat this limit.

    SEE: https://github.com/BerriAI/litellm/issues/9792
    """
    litellm.litellm_core_utils.logging_callback_manager.LoggingCallbackManager.MAX_CALLBACKS = (
        value
    )


update_litellm_max_callbacks()

[gustavz]

@krrishdholakia this issue still persists, how to prevent duplicate callbacks being added when doing it once per app lifecycle like this

litellm.callbacks = [MyAsyncCallback()]

[achpalaman]

@krrishdholakia @ishaan-berri @ishaan-jaff -- can y'all please take a look at this one? 🙏🏻 It is flooding our logs, and I'm unsure if it's causing issues (efficiency, or otherwise) on the LiteLLM side, so uncomfortable suppressing the log message without an RCA. It's been open for a bit and a bunch of folks are running into it as well...

Note: I am not adding ANY custom callbacks at atll, which is why I'm unsure why this is being triggered in the first place...

[gustavz]

@achpalaman we found the issue and had to reimplement our router logic. TLDR: You may always only ever create a single litellm.Router instance in a running app

Long version: litellm uses global lists to manage callbacks (among other things), and each Router instance will append new (duplicated) callbacks (default, not custom callbacks) to the global callbacks list upon instantiation. This means your app will ultimately break if you create multiple routers, or to put it different: You may always only ever create a single router instance in a running app, otherwise you will spam callbacks and likely also introduce other memory leaks (our pods always oomed running litellm clients for a while). Using and appending to global lists across instances is really some of the worst patterns I have ever come across, but maybe it's deliberate 🤷 .

[willbelr]

@achpalaman we found the issue and had to reimplement our router logic. TLDR: You may always only ever create a single litellm.Router instance in a running app

Is it the intented behavior ? Feels more like a bug or limitation.

Edit: For anyone wondering, this approach works around the issue;

@@ -33,6 +33,7 @@ class LiteAPI(QtCore.QObject):
         self.connectAttempts = 0
         self.initializing = False
         self.running = False
+        self.router = litellm.Router()
 
     @QtCore.pyqtSlot(str, dict, object)
     def run(self, prompt: str, parameters: dict, path: Path):
@@ -186,14 +187,15 @@ class LiteAPI(QtCore.QObject):
 
             mode = "image" if path else "text"
             fallback = self.core.adb[mode]["fallback"]
-            router = litellm.Router(
+            self.router.set_model_list(
                 model_list=[
                     {"model_name": m, "litellm_params": {"model": m, **parameters}}
                     for m in [model, fallback] if m
                 ]
             )
 
-            payload = router.completion(
+            payload = self.router.completion(
                 model=model,
                 fallbacks=[fallback],
                 messages=[{"content": content, "role": "user"}],