server-sdk-go
server-sdk-go copied to clipboard
livekit
Enable gstreamer as an ingestion source
We could make it easier to publish a track from any GStreamer source by providing an app src handler.
Hi,
I would be interested in broadcasting a video track coming from an external GStreamer application (written in C) to browsers through LiveKit.
How can I help? Can you give me some pointers on where to start digging?
Thanks for your help and for your work on LiveKit.
@Pyrrvs that'd be fantastic! I was thinking something like this, where we could allow users to construct the gstreamer pipeline. We would then read the output from appsink and publish it into a livekit room
In your use case, how is the video stream encoded/formatted?
Ok, I'll start looking at it this weekend!
In my case, the video stream is currently an H264 stream muxed in a mpegts container with an AAC audio track. It is then sent as an RTP stream. I have complete control over the source code as it's an in-house development we made using GStreamer. So I can easily change the audio encoder and get rid of the mpeg-ts muxer.
As my streamer is written in C/ C++ and LiveKit has no C/C++ SDK, I have 3 main leads to get my streams delivered through LiveKit:
- Develop a custom C/C++ LiveKit GStreamer sink that would take care of registering to the LiveKit instance and to send the packets. Using the LiveKit SDK through a Go to C binding based on something like this.
- Embed the heavy GStreamer-related lifting into GStreamer bins, rewrite my streamer in Go and re-use the bins components through a CGo binding or by constructing the pipeline as you suggested. It'll allow me to use the Go SDK directly.
- Write a C or C++ client SDK.
Either way I think this issue could be a good exercise to get to know the LiveKit code base.
Yeah, that seems right. We have company that's doing something that might be an option 4:
use webrtcbin in GStreamer as a sink.. then create a signal proxy with the Go SDK, i.e. let signaling to be handled by the Go SDK, but when it needs to publish a stream and generate an offer, use the offer created by webrtcbin and send that to the server. LiveKit can accept webrtc streams, but there's a good amount of signaling logic that takes place.
@davidzhao I tested this use case. webrtcbin is a very crude solution and works unstable. Here, too, we came to this conclusion. https://medium.com/tribe-staff/down-the-mcu-road-with-gstreamer-webrtc-d1df48a93492
@davidzhao I tested this use case. webrtcbin is a very crude solution and works unstable. Here, too, we came to this conclusion. https://medium.com/tribe-staff/down-the-mcu-road-with-gstreamer-webrtc-d1df48a93492
The date of the article you linked is circa pre-release gstreamer 1.14 when the webrtc element was just being introduced. There's been a lot of work on its webrtc stack since then.
@Pyrrvs that'd be fantastic! I was thinking something like this, where we could allow users to construct the gstreamer pipeline. We would then read the output from appsink and publish it into a livekit room
This would be a super useful feature for us. I am working on an embedded application with three cameras feeding into an nvidia jetson AGX. We have two gstreamer pipelines going:
- Feed the raw video into a loopback device for playback in the local browser
- Encode the video to h264 and record them to disk
We are planning on using livekit to stream the feeds over the internet so we can view them remotely. Right now, I am using the javascript SDK in the local browser to tap into the loopback device, but it would be preferable to tap into the recorder's existing h264 pipeline.
use
webrtcbinin GStreamer as a sink.. then create a signal proxy with the Go SDK, i.e. let signaling to be handled by the Go SDK, but when it needs to publish a stream and generate an offer, use the offer created by webrtcbin and send that to the server. LiveKit can accept webrtc streams, but there's a good amount of signaling logic that takes place.
I'm happy to try this out, but do you have any example code? I'm at a loss where to begin.
I'm happy to try this out, but do you have any example code? I'm at a loss where to begin. Hi @Pyrrvs, see https://github.com/centricular/webrtcsink and https://github.com/centricular/gstwebrtc-demos
I was more referring to example code for this:
let signaling to be handled by the Go SDK, but when it needs to publish a stream and generate an offer, use the offer created by webrtcbin and send that to the server
Our use case is very similar to that of @didiersmith. This would be a very very welcome feature.
Just my 2 cents, I personally feel that go-gst adds complexity to the SDK as we're now introducing CGO, and compilation time will be much longer. I made a PR recently that got approved; the crux of it is to enable any io.ReadCloser implementation as an ingress source. For livekit-cli, there's a PR to enable reading from Unix sockets so a Python process can publish some RTSP feed to that socket and LiveKit CLI will forward the data to the room.
In relation to this issue, I think it's possible to use GStreamer's shmsink element to publish to the Unix socket, and then in your app, you can do something like lksdk.NewLocalReaderTrack(unixReader, mime, opts...) where unixReader will listen to that socket. If you need more advice on this, you can PM me on Slack.
In summary, I personally believe the burden of ingesting from GStreamer should be handled by the custom application rather than the SDK. The SDK should just be a proxy for the custom apps to communicate with the LiveKit ecosystem.
@dwirya I agree with you. this seem to add too much complexity baking everything into the SDK. It makes sense to rely on the unix socket approach that you worked on. Let's get that PR merged and we can close this one.
