live-kit-cli: vp8 TCP input gives "error"="IVF signature mismatch"

[neilyoung]

Hi folks,

as shown in this thread it is perfectly possible to publish H.264 NAL units from a GStreamer pipeline to livekit-server via livekit-cli:

https://livekit-users.slack.com/archives/C01KVTJH6BX/p1700642085369689?thread_ts=1700500341.249709&cid=C01KVTJH6BX

The same seems to be not possible, since livekit-cli fires

2023/11/23 10:07:51 "msg"="Could not start writing" "error"="IVF signature mismatch"

Gstreamer Pipeline:

gst-launch-1.0 avfvideosrc device-index=0 ! video/x-raw, width=1280, height=720, framerate=30/1 ! videoconvert ! vp8enc deadline=1 ! tcpserversink port=16400

livekit-cli call:

livekit-cli join-room --url ws://localhost:7880 --api-key devkey --api-secret secret --room abcde --identity bot --publish vp8://localhost:16400

[neilyoung]

Shit. Duplicate of https://github.com/livekit/livekit-cli/issues/104

No solution :(

[neilyoung]

The expected header is in the first packet sent, if "udpsink" is used. But not in the "tcpserversink" payload...

But I guess lifekit is unable to publish udpsinks (?)

[neilyoung]

OK, found it. I have to be incredibly fast between launching the GST pipeline and starting the livekit-cli. Otherwise obviously the header buffer gets lost due to an internal buffer overflow in GST.

This pipeline works, but the client needs to be started within one second after the launch of this:

gst-launch-1.0 avfvideosrc device-index=0 ! video/x-raw, width=1280, height=720, framerate=30/1 ! videoconvert ! vp8enc ! avmux_ivf ! queue ! tcpserversink host=0.0.0.0 port=16400

Client:

livekit-cli join-room --url ws://localhost:7880 --api-key devkey --api-secret secret --room abcde --identity bot --publish vp8://127.0.0.1:16400

Anyway, quality is bad, latency is high. 720p @ 30fps, but just 400 kBbs. Maybe there is an additional setting for the bitrate at vp8enc.

EDIT: Latency decreases over time

EDIT2: Perfect results with this:

gst-launch-1.0 avfvideosrc device-index=0 ! video/x-raw, width=1280, height=720, framerate=30/1 ! videoconvert ! vp8enc deadline=1 target-bitrate=5000000 ! avmux_ivf ! queue ! tcpserversink host=0.0.0.0 port=16400