[Bug]: Facing deadlocks on LiteLLM

[xyannie]

What happened?

Version : v1.52.10

Ran some load tests which will hit litellm endpoint , observed occasionally will hit the following error

Relevant log output

ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError { code: "40P01", message: "deadlock detected", severity: "ERROR", detail: Some("Process 4199 waits for ShareLock on transaction 29865236; blocked by process 4225.
Process 4225 waits for ShareLock on transaction 29865235; blocked by process 4199."), column: None, hint: Some("See server log for query details.") }), transient: false })

Twitter / LinkedIn details

No response

[krrishdholakia]

ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError { code: "40P01", message: "deadlock detected", severity: "ERROR", detail: Some("Process 4199 waits for ShareLock on transaction 29865236; blocked by process 4225.

this deadlock seems to be from the prisma client writing to the db

can you please share steps to repro?

[xyannie]

I'll get back to you on litellm application side, but I conducted a load test on our GenAI models to assess the current throughput and thresholds. In our case, the workflow involves an agent invoking multiple tools (which may include additional LLM calls) to generate a final output.

Do let me know if there any additional information you need on our end.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[Bharat23]

We are facing a similar issue during high load

Message: DB read/write call failed: Error occurred during query execution:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError { code: "40P01", message: "deadlock detected", severity: "ERROR", detail: Some("Process 4189 waits for ShareLock on transaction 24394484; blocked by process 4194.\nProcess 4194 waits for ShareLock on transaction 24394483; blocked by process 4189."), column: None, hint: Some("See server log for query details.") }), transient: false })[Non-Blocking]LiteLLM Prisma Client Exception - update spend logs: Error occurred during query execution:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError { code: "40P01", message: "deadlock detected", severity: "ERROR", detail: Some("Process 4189 waits for ShareLock on transaction 24394484; blocked by process 4194.\nProcess 4194 waits for ShareLock on transaction 24394483; blocked by process 4189."), column: None, hint: Some("See server log for query details.") }), transient: false })
Traceback (most recent call last):
  File "/home/litellm/venv/lib/python3.13/site-packages/litellm/proxy/db/db_spend_update_writer.py", line 862, in _update_daily_spend
    async with prisma_client.db.batch_() as batcher:
               ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/litellm/venv/lib/python3.13/site-packages/prisma/client.py", line 478, in __aexit__
    await self.commit()
  File "/home/litellm/venv/lib/python3.13/site-packages/prisma/client.py", line 454,

[neubig]

Same here, we're specifically getting this on File "/usr/lib/python3.13/site-packages/litellm/proxy/db/db_spend_update_writer.py", line 862, in _update_daily_spend