livekit
Running agents with connect --room causes worker to receive sometimes two job requests instead of one
Hello! I am running simple livekit worker with command python main.py connect --room tpapaj-room (where tpapaj-room can be anything) using function
from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, llm
(...)
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
where entrypoint is my function where I create VoiceAssistant etc.
Sometimes it works correctly and one job request is received:
2024-11-09 17:33:47,917 - INFO livekit.agents - starting worker {"version": "0.10.1", "rtc-version": "0.18.0"}
2024-11-09 17:33:48,258 - INFO livekit.agents - registered worker {"id": "AW_xxxWY4", "region": "Germany", "protocol": 15, "node_id": "NC_xxxF3R"}
2024-11-09 17:33:48,259 - INFO livekit.agents - connecting to room tpapaj-room
2024-11-09 17:33:48,526 - INFO livekit.agents - received job request {"job_id": "AJ_xxxiSW", "dispatch_id": "", "room_name": "tpapaj-room", "agent_name": "", "resuming": false}
Sometimes on startup worker receives two job requests:
2024-11-09 17:34:15,144 - INFO livekit.agents - starting worker {"version": "0.10.1", "rtc-version": "0.18.0"}
2024-11-09 17:34:15,505 - INFO livekit.agents - registered worker {"id": "AW_yxxxWiS", "region": "Germany", "protocol": 15, "node_id": "NC_xxx39j"}
2024-11-09 17:34:15,507 - INFO livekit.agents - connecting to room tpapaj-room
2024-11-09 17:34:16,473 - INFO livekit.agents - received job request {"job_id": "AJ_xxxvAt", "dispatch_id": "", "room_name": "tpapaj-room", "agent_name": "", "resuming": false}
2024-11-09 17:34:16,581 - INFO livekit.agents - received job request {"job_id": "AJ_xxxPpE", "dispatch_id": "", "room_name": "tpapaj-room", "agent_name": "", "resuming": false}
This causes application to create two agents(entrypoint function is called twice) who then start talk to each other in a room. As you can see, received job requests have different ids(I masked parts of ids with xxx strings, just in case).
It looks like it has tendency to create two agents when --room argument uses a room name that was not used for a while and restarting the worker with the same room usually works correctly and creates a single agent.
I expected to see just one job request being sent to the worker.
It seems to be similar issue to https://github.com/livekit/agents/issues/759 but it is claimed there that the problem was fixed, but I still see the problem. I tried livekit-agent versions 0.10.1, 0.10.2, 0.11.1 without success.
Current livekit packages in pip freeze:
livekit==0.18.0
livekit-agents==0.11.1
livekit-api==0.7.1
livekit-plugins-elevenlabs==0.7.7
livekit-plugins-openai==0.10.5
livekit-plugins-silero==0.7.3
livekit-protocol==0.6.0
Also seeing the same issue.
A workaround I found: filter requests with a custom request_fnc
seen = set()
async def request_fnc(req: JobRequest):
if req.room.name != room_id or req.room.name in seen:
await req.reject()
return
seen.add(req.room.name)
# accept the job request
await req.accept(
...
)
opts = WorkerOptions(
...
request_fnc=request_fnc,
)
but the issue then becomes the duplicate request will be repeatedly attempted with the worker despite being rejected except the first:
2024-12-16 16:23:56,193 - INFO livekit.agents - starting worker {"version": "0.12.2", "rtc-version": "0.18.2"}
2024-12-16 16:23:56,245 - INFO livekit.agents - registered worker {"id": "AW_tiTubQiPGdSP", "region": "US East", "protocol": 15, "node_id": "NC_OASHBURN1A_reKoqp3EoNE2"}
2024-12-16 16:23:56,245 - INFO livekit.agents - connecting to room playground-2948-Cekl
2024-12-16 16:23:56,578 - INFO livekit.agents - received job request {"job_id": "AJ_CNAniTzqabmp", "dispatch_id": "", "room_name": "playground-2948-Cekl", "agent_name": "", "resuming": false}
2024-12-16 16:23:56 [info ] Accepting request for room: playground-2948-Cekl name=livekit-framework-worker
2024-12-16 16:23:56,584 - INFO livekit.agents - received job request {"job_id": "AJ_mdFTSoccBjGJ", "dispatch_id": "", "room_name": "playground-2948-Cekl", "agent_name": "", "resuming": false}
2024-12-16 16:23:56,598 - INFO livekit.agents - received job request {"job_id": "AJ_mdFTSoccBjGJ", "dispatch_id": "", "room_name": "playground-2948-Cekl", "agent_name": "", "resuming": false}
2024-12-16 16:23:56,603 - INFO livekit.agents - received job request {"job_id": "AJ_mdFTSoccBjGJ", "dispatch_id": "", "room_name": "playground-2948-Cekl", "agent_name": "", "resuming": false}
I am also facing the same bug, where for some reason when we perform explicit agent dispatch using create_dispatch, the running Worker starts two agent processes.
FYI, we've also seen this one. We coded around it with an in-memory cache of job request id's similar to ChenghaoMou's solution above.
Also seeing the same issue.
A workaround I found: filter requests with a custom
request_fncseen = set() async def request_fnc(req: JobRequest): if req.room.name != room_id or req.room.name in seen: await req.reject() return seen.add(req.room.name) # accept the job request await req.accept( ... )
opts = WorkerOptions( ... request_fnc=request_fnc, ) but the issue then becomes the duplicate request will be repeatedly attempted with the worker despite being rejected except the first:
2024-12-16 16:23:56,193 - INFO livekit.agents - starting worker {"version": "0.12.2", "rtc-version": "0.18.2"} 2024-12-16 16:23:56,245 - INFO livekit.agents - registered worker {"id": "AW_tiTubQiPGdSP", "region": "US East", "protocol": 15, "node_id": "NC_OASHBURN1A_reKoqp3EoNE2"} 2024-12-16 16:23:56,245 - INFO livekit.agents - connecting to room playground-2948-Cekl 2024-12-16 16:23:56,578 - INFO livekit.agents - received job request {"job_id": "AJ_CNAniTzqabmp", "dispatch_id": "", "room_name": "playground-2948-Cekl", "agent_name": "", "resuming": false} 2024-12-16 16:23:56 [info ] Accepting request for room: playground-2948-Cekl name=livekit-framework-worker 2024-12-16 16:23:56,584 - INFO livekit.agents - received job request {"job_id": "AJ_mdFTSoccBjGJ", "dispatch_id": "", "room_name": "playground-2948-Cekl", "agent_name": "", "resuming": false} 2024-12-16 16:23:56,598 - INFO livekit.agents - received job request {"job_id": "AJ_mdFTSoccBjGJ", "dispatch_id": "", "room_name": "playground-2948-Cekl", "agent_name": "", "resuming": false} 2024-12-16 16:23:56,603 - INFO livekit.agents - received job request {"job_id": "AJ_mdFTSoccBjGJ", "dispatch_id": "", "room_name": "playground-2948-Cekl", "agent_name": "", "resuming": false}
It's from the documentation
If the request is rejected, it's sent to the next available worker.
So I think the best approach is just to return in the entrypoint function.
We experienced the same issue and had to work around it. any fix in the future?
