Multiple avatars do not switch correctly across agent sessions

[Viktoriagrg]

Bug Description

When running multiple agent sessions with different avatars, the system always uses the last started avatar, regardless of which session is active. So I can make a conclusion that there is no proper connection between agent sessions and their assigned avatars.

As a result:

• Starting multiple sessions with distinct avatars does not produce separate visual representations.
• Switching between sessions does not update the avatar to the one assigned to that session.

Expected Behavior

• Each agent session should maintain its own avatar.
• Switching between sessions should correctly display the corresponding avatar for that session.

Reproduction Steps

Define multiple AgentSession instances.
Assign a different video avatar to each session.
Start all sessions.
Observe that every session displays the last started avatar instead of the assigned one.

Operating System

macOS

Models Used

No response

Package Versions

livekit==1.0.20
livekit-agents==1.3.6

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response

[longcw]

it's designed to have one AgentSession with multiple Agent, could you explain a bit why you want to start multiple sessions and how you did that?

regarding the avatar, every time you start an avatar session, it sends a request to the avatar provider to start an avatar and join to the room, so if you started multiple agent session and avatars, you will need to specify a different avatar participant id in the avatar plugin, and there might be multiple avatars in the room.

[Viktoriagrg]

Hello, I start multiple sessions to assign each agent a separate avatar. However, I couldn’t find any logic for configuring avatars for each agent in your repository or documentation. I did find some information about the session ID. Could you provide a code reference for the logic mentioned? As I understand, you can set the avatar_participant_identity and/or avatar_participant_name, and each agent can have a different avatar—correct? I tried this, but it doesn’t work as expected.

For multiple sessions, I use the following code sample:

        await intake_avatar.start(intake_session, room=ctx.room)
        await intake_session.start(
            room=ctx.room,
            agent=IntakeAgent(),
            room_options=room_options,
        )
        
        await background_audio.start(room=ctx.room, agent_session=intake_session)
        
        await intake_session.generate_reply(
            instructions="Greet the user and offer your assistance."
        )
 
        await userdata.handoff_event.wait()
        
        if userdata.next_agent == "customer_service":
            await intake_session.aclose()
            await asyncio.sleep(0.5)
            userdata.handoff_event.clear()
            userdata.next_agent = "billing"
   
            await customer_service_avatar.start(customer_service_session, room=ctx.room)
            await customer_service_session.start(
                room=ctx.room,
                agent=CustomerServiceAgent(),
                room_options=room_options,
            )
            
            await userdata.handoff_event.wait()
            
            if userdata.next_agent == "billing":
                await customer_service_session.aclose()
                await asyncio.sleep(0.5)
                await billing_avatar.start(billing_session, room=ctx.room)
                await billing_session.start(
                    room=ctx.room,
                    agent=BillingAgent(),
                    room_options=room_options,
                )
                

With this method, avatars switch as expected. However, when I try starting all sessions first and then assigning avatars, like this:

await intake_avatar.start(intake_session, room=ctx.room)
await customer_service_avatar.start(customer_service_session, room=ctx.room)
await billing_avatar.start(billing_session, room=ctx.room)
await intake_session.start(
            room=ctx.room,
            agent=IntakeAgent(),
            room_options=room_options,
        )
await customer_service_session.start(
                room=ctx.room,
                agent=CustomerServiceAgent(),
                room_options=room_options,
            )
await billing_session.start(
                    room=ctx.room,
                    agent=BillingAgent(),
                    room_options=room_options,
                )

Only the last avatar shows for all sessions.

Ultimately, I want to have multiple agents, each with a unique avatar in the same room, along with an orchestrator agent.

[dimasimonowich]

@longcw Any updates on this? Got the same question as above.

[Arslan-Mehmood1]

Is there any guideline on how to create custom Avatar plugin and integrate to Livekit?

[chenghao-mou]

As mentioned by @longcw, there is only one active AgentSession at any given time, regardless of how many agents you have. So when you tried "starting all sessions first", you would only have the last session activated.

This is very similar to GIL in Python to avoid issues like agents talking over each other or complexities and burdens to orchestrate them not to. With that being said, we are exploring something like background agents so that they can be active without affecting each other.

[Viktoriagrg]

@chenghao-mou @longcw I understand your point. However, I’m encountering another issue with avatar switching when changing agents within the same agent session. Details are described here #4315