Running Multiple Agent Workers for Different Use Cases (Voice Assistant vs Virtual Avatar)

[ahenawy]

trafficstars

I’m working on a system that uses LiveKit Agents for two distinct purposes:

• A voice assistant
• A virtual avatar (e.g., animated character responding with custom behavior)

From the Agent Dispatch documentation, I understand that I can use a dispatch function to route requests to different worker instances. I want to confirm my understanding of the correct approach for running multiple agents for different purposes.

@router.get("/getToken")
async def getToken(user_id: str, user_name: str, agent_name: str = "voice-agent"):
    token = (
        AccessToken(os.getenv("LIVEKIT_API_KEY"), os.getenv("LIVEKIT_API_SECRET"))
        .with_grants(
            VideoGrants(
                room_join=True,
                room=f"room_{shortuuid()}",
            )
        )
        .with_room_config(
            RoomConfiguration(
                agents=[
                    RoomAgentDispatch(
                        agent_name=agent_name
                    ),
                ],
            ),
        )
    )
    return token.to_jwt()

    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            worker_type=WorkerType.ROOM,
            agent_name="avatar-agent",
        ),
    )

Is this the recommended way to run multiple workers with distinct behaviors?

[theomonnom]

Yes, exactly — using agent_name is the right approach!

cc @bcherry, we might want to consider improving the documentation to make this clearer.

[ahenawy]

I have two LiveKit workers configured as follows:

cli.run_app(
    WorkerOptions(
        entrypoint_fnc=entrypoint,
        agent_name="virtual-avatar",  # Unique name for explicit dispatch
    )
)

cli.run_app(
    WorkerOptions(
        entrypoint_fnc=entrypoint,
        agent_name="voice-assistant",  # Unique name for explicit dispatch
    )
)

Despite this setup, the first worker that starts accepts the job request and joins the room, ignoring the room configuration and the RoomAgentDispatch specified in the generated token.

[bcherry]

@ahenawy yes you've set it up correctly. the current result you describe does sound like a bug. are you starting your workers with python <worker-file>.py dev?

[ahenawy]

@bcherry Yes, I do for both workers. One of them always accepted the job, ignoring the agent's name. Can you please mark it as a bug?

[pavan-aubergine]

@bcherry Yes, I do for both workers. One of them always accepted the job, ignoring the agent's name. Can you please mark it as a bug?

Hey there @ahenawy , I am facing a similar issue for setting up different agents based on different scenario. Did you find the solution for this?

[davidzhao]

@pavan-aubergine can you describe what you are seeing? we have high confidence that this works as intended. our own environments/demo agents are dispatched this way

[pavan-aubergine]

Sure @davidzhao , Below is the code snippet of my workerfile.py.

I have created:

• 2 different entrypoint functions: One's a multi-user-transcriber & Another's a Voice Agent
• I have also added 2 request_fnc for the scenarios as per documentation: [https://docs.livekit.io/agents/worker/options/#request-handler]
• I have also added the agent name, in worker_options.

I am developing with the livekit agents playground where I am able to connect to room & ai-interview-mode agent is being connected but manual-interview-mode agent never connects( in different scenarios as I only need either of them called.

The code is able to get into the AI interview request fnc and handle the session confirmed it by logs, but the manual interview agent is never being dispatched(no request function logs neither entrypoint logs). Even job request is not received in logs for the manual interview

async def request_fnc(req: JobRequest):
    print('++++++++++++++++++++++++++')
    print('AI interview mode request received in request_fnc')
    print('++++++++++++++++++++++++++')
    # accept the job request
    await req.accept(
        # the agent's name (Participant.name), defaults to ""
        name="ai-interview-agent",
        # the agent's identity (Participant.identity), defaults to "agent-<jobid>"
        # identity="identity",
        # attributes to set on the agent participant upon join
        # attributes={"myagent": "rocks"},
    )

    # or reject it
    # await req.reject()

async def manual_interview_request_fnc(req: JobRequest):
    print('++++++++++++++++++++++++++')
    print('Manual interview mode request received in request_fnc')
    print('++++++++++++++++++++++++++')
    # accept the job request
    await req.accept(
        # the agent's name (Participant.name), defaults to ""
        name="manual-interview-agent",
        # the agent's identity (Participant.identity), defaults to "agent-<jobid>"
        # identity="manual-interview-agent",
        # attributes to set on the agent participant upon join
        # attributes={"myagent": "rocks"},
    )

    # or reject it
    # await req.reject()

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=manual_interview_entrypoint, prewarm_fnc=prewarm, agent_name="manual-interview-agent", request_fnc=manual_interview_request_fnc))
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name="ai-interview-agent", request_fnc=request_fnc))

API Code:

manual_interview_token = api.AccessToken(api_key, api_secret) \
                .with_identity(f"host_{uuid.uuid4().hex[:8]}") \
                .with_name(request.created_by) \
                .with_metadata(json.dumps(user_metadata)) \
                .with_grants(api.VideoGrants(
                    room_join=True,
                    room=request.room_name,
                    room_create=True,
                    room_admin=True
                ))\
                .with_room_config(
                RoomConfiguration(
                    agents=[
                        RoomAgentDispatch(agent_name="manual-interview-agent", metadata='{"user_id": "54321"}')
                    ],),)\
            .to_jwt()

ai_interview_token = api.AccessToken(api_key, api_secret) \
            .with_identity(f"participant_{uuid.uuid4().hex[:8]}") \
            .with_name(request.employee_name) \
            .with_metadata(json.dumps(user_metadata)) \
            .with_grants(api.VideoGrants(
                room_join=True,
                room=request.room_name,
            ))\
            .with_room_config(
                RoomConfiguration(
                    agents=[
                        RoomAgentDispatch(agent_name="ai-interview-agent", metadata='{"user_id": "12345"}')
                    ],),)\
            .to_jwt()

[longcw]

You cannot run two run_app like the following, you needs to run them as two python scripts, or using multiprocessing or subprocesses in python.

cli.run_app(WorkerOptions(entrypoint_fnc=manual_interview_entrypoint, prewarm_fnc=prewarm, agent_name="manual-interview-agent", request_fnc=manual_interview_request_fnc))
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name="ai-interview-agent", request_fnc=request_fnc))

[pavan-aubergine]

I am able to run now. Thanks for the help, @longcw @davidzhao !

I think adding an example of How to call required agent(s) based on condition would give developers better clarity.

[antonioallen]

How did you solve @pavan-aubergine, how did you specify multiple workers?

[pavan-aubergine]

How did you solve @pavan-aubergine, how did you specify multiple workers?

Hey @antonioallen , I was able to achieve the same with the last comment of longcw, I believe they should have a proper documentation of how to handle scenario involving multiple agents & different entrypoints, It was missing at the time I was developing the application using livekit.

[Taymed]

is there any documentation, i'm looking to implement a solution to spin new agents to life that would be reachable through dedicated phone numbers

[linear[bot]]

from @bcherry:

Yes the docs are accurate. Run your agents as separate programs with different agent names then setup your inbound SIP trunks - one per number - with a dispatch rule for the agent you want for that number

[Taymed]

can i have some links please