livekit
got errors while building up avatar with bithuman like "process memory usage is high" and "worker is at full capacity
I want to use avatar with bithuman, but got "process memory usage is high" and "worker is at full capacity, marking as unavailable". Do you have any advice?
2025-05-19 14:18:27.262 | INFO | bithuman.video_graph.navigator:from_workspace:132 - Loading model from /home/ubuntu/albert_einstein.imx 2025-05-19 14:18:27.791 | INFO | bithuman.video_graph.video_script:update_runtime_configs:334 - Updated runtime configs from model: {} 2025-05-19 14:18:27.794 | INFO | bithuman.runtime:load_data:408 - Loading model data: 1 models and 0 fillers 2025-05-19 14:18:29,878 - livekit.agents - WARNING - process memory usage is high {"message": "process memory usage is high", "level": "WARNING", "name": "livekit.agents", "memory_usage_mb": 582.828125, "memory_warn_mb": 500, "memory_limit_mb": 0, "pid": 3526106, "job_id": "XXX", "timestamp": "2025-05-19T06:18:29.878112+00:00"} 2025-05-19 14:18:32.269 | INFO | bithuman.runtime:load_data:443 - Model data loaded successfully 2025-05-19 14:18:32,669 - livekit.agents - INFO - worker is at full capacity, marking as unavailable {"message": "worker is at full capacity, marking as unavailable", "level": "INFO", "name": "livekit.agents", "load": 0.8388, "threshold": "_WorkerEnvOption(dev_default=inf, prod_default=0.75)", "timestamp": "2025-05-19T06:18:32.669676+00:00"}
Are you running in Docker? What is the system information and avatar worker setup?
The process memory usage is high is a warning to indicate the agent is using more RAM than usual, but given you are using an avatar model it's expected, you can increase the warning threshold as in this example.
worker is at full capacity, marking as unavailable means the worker will not accept new jobs since the cpu usage is high at the moment. The started job should still running. By default it's based on the cpu usage with threshold 75% in production mode.
The
process memory usage is highis a warning to indicate the agent is using more RAM than usual, but given you are using an avatar model it's expected, you can increase the warning threshold as in this example.
worker is at full capacity, marking as unavailablemeans the worker will not accept new jobs since the cpu usage is high at the moment. The started job should still running. By default it's based on the cpu usage with threshold 75% in production mode.
Thanks @longcw ， I will try to increase the warning threshold.
Not sure if this helps you but I had severe performance issues on AWS when building the Docker image for linux/amd64. After changing it to linux/arm64 that completely went away.
