pion
High CPU usage when connecting
Your environment.
- Version: v4.0.7
- Browser: Firefox 138 (should be irrelevant though)
- Other Information - https://github.com/livekit/livekit/issues/3316
What did you do?
We run livekit (which uses pion) on a k8s cluster. Most of the data is already collected here, but here's a quick summary:
We run livekit with host_network: true to get UDP working without a UDP load balancer/proxy. This means that the pod can see all network interfaces on the host, including interfaces of each other pod. However, it doesn't reload the list of available interfaces (often enough) so that when pods are stopped on k8s, it still tries to access their interface.
We have a large range of possible UDP ports (50000-60000), so this loop tries 10k times for every interface that went missing since the process started. This is done for every user that connects to livekit
As a workaround we use the "Includes" interface filter to only try to bind to the public interface. But to be more resilient against kernel/hardware updates I'd rather not do this pinning.
Unfortunately I don't know how to provide you with a runnable example of this. Does this (and the referenced) issue contain enough information to tackle this problem?
What did you expect?
CPU usage that corresponds to number of simultaneous connections/streams
What happened?
Since the update to v1.8.3 (which updated pion/ice from v2 to v4), we see high CPU spikes to up to 50% for single participants that don't even publish any streams yet
What behavior would you expect/prefer?
- pion/ice doesn't know what interface you would like to use.
- Can we determine if an interface is ALWAYS going to fail, or just a single port causes the failure?
Hi @Sean-Der thanks for your response!
pion/ice doesn't know what interface you would like to use
oh, I assumed pion/ice is the one loading the interface list, since it is the one using it. Can you help me pinpoint where the used interfaces are loaded? That part of the code should probably check for new/removed interfaces from time to time
Can we determine i fan interface is ALWAYS going to fail
in our case, checking if the interface still exists would do the job. Given your first point, I'm not sure if pion/ice is the place to do this check though