egress
egress copied to clipboard
Published
20 hours ago •
livekit
livekit
[BUG] Egress jobs not using S3 storage as defined in service config by default
Describe the bug In the egress YAML configuration I've set the s3 configuration settings per the documentation:
redis:
address: localhost:6379
api_key: xxxxxxxxx
api_secret: yyyyyyyyyyy
ws_url: wss://livekit.example.com
health_port: 11010
prometheus_port: 11011
s3:
access_key: AKxxxxxxxxxxxxxxxx
secret: verySecret
bucket: livekit-bucket
region: us-east-1
When I submit an egress request without storage configuration my recording files are saved locally. If I include the s3 configuration in the request the recoding files are saved to s3.
Note in the below the local storage attempt fails due to permissions issues and is expected.
Egress Version v1.9.0 (docker: livekit/egress@sha256:d62f515668d56df24082ec722a7a78134bc14ff331a2c0402ac90e8fe0fe0067)
Egress Request
Failed request
cat << EOF > request.json
{
"room_name": "test-room",
"layout": "grid",
"file": {
"file_type": "MP4",
"filepath": "test/test-room.mp4"
}
}
EOF
lk egress start --type room-composite ./request.json
Log entries:
2025-01-17T18:08:10.664Z INFO egress server/server_rpc.go:58 request received {"nodeID": "NE_X9tH35HcQqWr", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:08:10.664Z INFO egress server/server_rpc.go:68 request validated {"nodeID": "NE_X9tH35HcQqWr", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA", "requestType": "room_composite", "outputType": "file", "room": "test-room", "request": {"RoomComposite":{"room_name":"test-room","layout":"grid","Output":{"File":{"file_type":1,"filepath":"test/test-room.mp4","Output":null}},"Options":null}}}
2025-01-17T18:08:10.691Z INFO egress redis/redis.go:142 connecting to redis {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA", "simple": true, "addr": "localhost:6379"}
2025-01-17T18:08:10.765Z INFO egress source/web.go:150 xvfb: The XKEYBOARD keymap compiler (xkbcomp) reports:
> Warning: Could not resolve keysym XF86CameraAccessEnable
> Warning: Could not resolve keysym XF86CameraAccessDisable
> Warning: Could not resolve keysym XF86CameraAccessToggle
> Warning: Could not resolve keysym XF86NextElement
> Warning: Could not resolve keysym XF86PreviousElement
> Warning: Could not resolve keysym XF86AutopilotEngageToggle
> Warning: Could not resolve keysym XF86MarkWaypoint
> {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:08:10.765Z INFO egress source/web.go:150 xvfb: Warning: Could not resolve keysym XF86Sos
> Warning: Could not resolve keysym XF86NavChart
> Warning: Could not resolve keysym XF86FishingChart
> Warning: Could not resolve keysym XF86SingleRangeRadar
> Warning: Could not resolve keysym XF86DualRangeRadar
> Warning: Could not resolve keysym XF86RadarOverlay
> Warning: Could not resolve keysym XF86TraditionalSonar
> Warning: Could not resolve keysym XF86ClearvuSonar
> Warning: Could not resolve keysym XF86SidevuSonar
> Warning: Could not resolve keysym XF86NavInfo
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:08:10.767Z INFO egress source/web.go:150 xvfb: Errors from xkbcomp are not fatal to the X server
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:08:12.630Z INFO egress source/web.go:288 chrome: START_RECORDING {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:08:13.565Z INFO egress pipeline/watch.go:252 pipeline playing {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:08:13.568Z INFO egress info/io.go:178 egress_active {"nodeID": "NE_X9tH35HcQqWr", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA", "requestType": "room_composite", "outputType": "file", "error": "", "code": 0, "details": ""}
2025-01-17T18:09:27.639Z INFO egress info/io.go:178 egress_ending {"nodeID": "NE_X9tH35HcQqWr", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA", "requestType": "room_composite", "outputType": "file", "error": "", "code": 0, "details": "End reason: StopEgress API"}
2025-01-17T18:09:27.756Z INFO egress pipeline/watch.go:135 EOS received {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:09:27.831Z INFO egress source/web.go:150 xvfb: The XKEYBOARD keymap compiler (xkbcomp) reports:
> Warning: Could not resolve keysym XF86CameraAccessEnable
> Warning: Could not resolve keysym XF86CameraAccessDisable
> Warning: Could not resolve keysym XF86CameraAccessToggle
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:09:27.831Z INFO egress source/web.go:150 xvfb: > Warning: Could not resolve keysym XF86NextElement
> Warning: Could not resolve keysym XF86PreviousElement
> Warning: Could not resolve keysym XF86AutopilotEngageToggle
> Warning: Could not resolve keysym XF86MarkWaypoint
> Warning: Could not resolve keysym XF86Sos
> Warning: Could not resolve keysym XF86NavChart
> Warning: Could not resolve keysym XF86FishingChart
> Warning: Could not resolve keysym XF86SingleRangeRadar
> {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:09:27.831Z INFO egress source/web.go:150 xvfb: Warning: Could not resolve keysym XF86DualRangeRadar
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:09:27.831Z INFO egress source/web.go:150 xvfb: > Warning: Could not resolve keysym XF86RadarOverlay
> Warning: Could not resolve keysym XF86TraditionalSonar
> Warning: {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:09:27.831Z INFO egress source/web.go:150 xvfb: Could not resolve keysym XF86ClearvuSonar
> Warning: Could not resolve keysym XF86SidevuSonar
> Warning: Could not resolve keysym XF86NavInfo
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:09:27.833Z INFO egress source/web.go:150 xvfb: Errors from xkbcomp are not fatal to the X server
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_ccBRAwCn9Uqp", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA"}
2025-01-17T18:09:27.858Z INFO egress info/io.go:178 egress_failed {"nodeID": "NE_X9tH35HcQqWr", "clusterID": "", "egressID": "EG_VdPDoBBMxTyA", "requestType": "room_composite", "outputType": "file", "error": "mkdir test/: permission denied", "code": 0, "details": "End reason: StopEgress API"}
Successful request
cat << EOF > request.json
{
"room_name": "test-room",
"layout": "grid",
"file": {
"file_type": "MP4",
"filepath": "test/test-room.mp4"
"s3": {
"access_key": "AKxxxxxxxxxxxxxxxx",
"secret": "verySecret",
"region": "us-east-1",
"bucket": "livekit-bucket"
}
}
}
EOF
lk egress start --type room-composite ./request.json
Log entries:
2025-01-17T17:44:57.381Z INFO egress server/server_rpc.go:58 request received {"nodeID": "NE_X9tH35HcQqWr", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn"}
2025-01-17T17:44:57.382Z INFO egress server/server_rpc.go:68 request validated {"nodeID": "NE_X9tH35HcQqWr", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn", "requestType": "room_composite", "outputType": "file", "room": "test-room", "request": {"RoomComposite":{"room_name":"test-room","layout":"grid","Output":{"File":{"file_type":1,"filepath":"test/test-room.mp4","Output":{"S3":{"access_key":"{access_key}","secret":"{secret}","region":"us-east-1","bucket":"livekit-bucket"}}}},"Options":null}}}
2025-01-17T17:44:57.405Z INFO egress redis/redis.go:142 connecting to redis {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_Q4stqBCJZfd6", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn", "simple": true, "addr": "localhost:6379"}
2025-01-17T17:44:57.471Z INFO egress source/web.go:150 xvfb: The XKEYBOARD keymap compiler (xkbcomp) reports:
> Warning: Could not resolve keysym XF86CameraAccessEnable
> Warning: Could not resolve keysym XF86CameraAccessDisable
> Warning: Could not resolve keysym XF86CameraAccessToggle
> Warning: Could not resolve keysym XF86NextElement
> Warning: Could not resolve keysym XF86PreviousElement
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_Q4stqBCJZfd6", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn"}
2025-01-17T17:44:57.472Z INFO egress source/web.go:150 xvfb: > Warning: Could not resolve keysym XF86AutopilotEngageToggle
> Warning: Could not resolve keysym XF86MarkWaypoint
> Warning: Could not resolve keysym XF86Sos
> Warning: Could not resolve keysym XF86NavChart
> Warning: Could not resolve keysym XF86FishingChart
> Warning: Could not resolve keysym XF86SingleRangeRadar
> Warning: Could not resolve keysym XF86DualRangeRadar
> Warning: Could not resolve keysym XF86RadarOverlay
> Warning: Could not resolve keysym XF86TraditionalSonar
> Warning: Could not resolve keysym XF86ClearvuSonar
> Warning: Could not resolve keysym XF86SidevuSonar
> Warning: Could not resolve keysym XF86NavInfo
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_Q4stqBCJZfd6", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn"}
2025-01-17T17:44:57.473Z INFO egress source/web.go:150 xvfb: Errors from xkbcomp are not fatal to the X server
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_Q4stqBCJZfd6", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn"}
2025-01-17T17:44:59.249Z INFO egress source/web.go:288 chrome: START_RECORDING {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_Q4stqBCJZfd6", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn"}
2025-01-17T17:45:00.280Z INFO egress pipeline/watch.go:252 pipeline playing {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_Q4stqBCJZfd6", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn"}
2025-01-17T17:45:00.283Z INFO egress info/io.go:178 egress_active {"nodeID": "NE_X9tH35HcQqWr", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn", "requestType": "room_composite", "outputType": "file", "error": "", "code": 0, "details": ""}
2025-01-17T17:45:32.433Z INFO egress info/io.go:178 egress_ending {"nodeID": "NE_X9tH35HcQqWr", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn", "requestType": "room_composite", "outputType": "file", "error": "", "code": 0, "details": "End reason: StopEgress API"}
2025-01-17T17:45:32.524Z INFO egress pipeline/watch.go:135 EOS received {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_Q4stqBCJZfd6", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn"}
2025-01-17T17:45:32.586Z INFO egress source/web.go:150 xvfb: The XKEYBOARD keymap compiler (xkbcomp) reports:
> Warning: Could not resolve keysym XF86CameraAccessEnable
> Warning: Could not resolve keysym XF86CameraAccessDisable
> Warning: Could not resolve keysym XF86CameraAccessToggle
> Warning: {"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_Q4stqBCJZfd6", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn"}
2025-01-17T17:45:32.587Z INFO egress source/web.go:150 xvfb: Could not resolve keysym XF86NextElement
> Warning: Could not resolve keysym XF86PreviousElement
> Warning: Could not resolve keysym XF86AutopilotEngageToggle
> Warning: Could not resolve keysym XF86MarkWaypoint
> Warning: Could not resolve keysym XF86Sos
> Warning: Could not resolve keysym XF86NavChart
> Warning: Could not resolve keysym XF86FishingChart
> Warning: Could not resolve keysym XF86SingleRangeRadar
> Warning: Could not resolve keysym XF86DualRangeRadar
> Warning: Could not resolve keysym XF86RadarOverlay
> Warning: Could not resolve keysym XF86TraditionalSonar
> Warning: Could not resolve keysym XF86ClearvuSonar
> Warning: Could not resolve keysym XF86SidevuSonar
> Warning: Could not resolve keysym XF86NavInfo
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_Q4stqBCJZfd6", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn"}
2025-01-17T17:45:32.588Z INFO egress source/web.go:150 xvfb: Errors from xkbcomp are not fatal to the X server
{"nodeID": "NE_X9tH35HcQqWr", "handlerID": "EGH_Q4stqBCJZfd6", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn"}
2025-01-17T17:45:33.461Z INFO egress info/io.go:178 egress_complete {"nodeID": "NE_X9tH35HcQqWr", "clusterID": "", "egressID": "EG_irUvQ8oBwzyn", "requestType": "room_composite", "outputType": "file", "error": "", "code": 0, "details": "End reason: StopEgress API"}
Additional context I've tried setting the s3 block to be empty but that results in a region not set error. If I set the region and bucket names I get a bad credential error.
I have similar logs. This situation occurs when I upgrade to v1.9.0. It was solved when I downgraded to v1.8.4.
I confirm. In 1.8.4, the default s3 configuration works if there is no s3 object in the request. In the latest version, you need to specify s3 explicitly.
@mexvod @furkanKotic @Malcolm-GetAHead This issue is due to the outdated README, see my PR: https://github.com/livekit/egress/pull/916
@frostbyte73 Sorry if im tagging the wrong person but this should be closed.