[Bug]: LLM Migration may enter deadlock when running multiple WFO instances

[Mark90]

Contact Details

No response

What happened?

When starting multiple WFO instances we just noticed that they would hang on this line:

Running LLM migration

Inspecting the open connections with

select pid,
       usename,
       pg_blocking_pids(pid) as blocked_by,
       query as blocked_query
from pg_stat_activity
where cardinality(pg_blocking_pids(pid)) > 0;

returned this:

Version

4.6.0rc4

What python version are you seeing the problem on?

Python 3.12

Relevant log output