Setting metadata on all participants causes errors

[moham96]

Describe the bug Setting metadata on all participants without delay causes failure, for testing I used server-sdk-js to call updateParticipant on all participants in the room. The server room has multiple participants simulated with livekit-cli load-test and when the below example is run it throws an exception because of 500 code from the twirp server:

import { RoomServiceClient } from "livekit-server-sdk";


const roomName = "Students";

const roomServiceClient = new RoomServiceClient(
  `http://127.0.0.1:7880`,
  "devkey",
  "secret"
);

roomServiceClient.listParticipants(roomName).then((participants) =>
  participants.forEach((participant) => {
    roomServiceClient.updateParticipant(
      roomName,
      participant.identity,
      String.fromCharCode(0 | (Math.random() * 26 + 97))
    );
  })
);
console.log("done");

Server

• Version: [1.4.4]
• Environment: local dev macos
• any other information about your deployment setup

Client

• SDK: js

To Reproduce Steps to reproduce the behavior:

1. simulate multiple participants using livekit-cli load-test command
2. Run the code above
3. You will get 500 server errors because the livekit server fails to set the metadata on some participants

Expected behavior I expect the metadata to be set even without delay between calls

Screenshots If applicable, add screenshots to help explain your problem.

Additional context I reported this previously here https://github.com/livekit/server-sdk-js/issues/79 but further debugging revealed that the failure is in the livekit server and not the SDK, note that rate limiting the calls does mitigate the issue but it is not ideal, and since the twirp interface used in the livekit server doesn't include rate limiting, there is something else that is causing the failure.

[davidzhao]

how many participants were in the room at that time?

[moham96]

how many participants were in the room at that time?

Well 40 but i don't think it is related to the number of participants but to the speed of sending the requests, but i could be wrong since i didn't test with lower number of participants

[davidzhao]

ok, you should throttle your requests. without throttling eventually you'll overwhelm the server and/or cause the server to drop requests. a general rule of thumb is no more than 5 concurrent requests at the same time.

[moham96]

ok, you should throttle your requests. without throttling eventually you'll overwhelm the server and/or cause the server to drop requests. a general rule of thumb is no more than 5 concurrent requests at the same time.

Yes but i had to throttle it considerably, like 100ms between each request which is not very appealing for my use case My app depend heavily on sending metadata and the UI reacts immediately to metadata changes, Having a delay of 1000ms is a lot for the ui to reflect these changes