[HLS] Support for MPEG-TS source streams

[lars18th]

Describe the feature

The regular mode to ingest a stream in rtsp-simple-server is to upload the audio and video substreams in different tracks over the RTSP protocol in the same session. So we're using a multiplexed connection with demuxed A/V streams.

However, this is not the only solution. We can ingest a stream directly muxed inside a MPEG-TS container. Here and exemple:

• ffmpeg -re -stream_loop -1 -i emptyvideo.mkv -c copy -f mpegts pipe:1 \ | gst-launch-1.0 -v fdsrc ! queue ! tsparse set-timestamps=true ! video/mpegts ! rtspclientsink protocols=tcp location=rtsp://127.0.0.1:8554/test/stream

In this example I'm using the regular command from ffmpeg to generate the test stream in a MPEG-TS container. And then using the gstreamer to ingest it to the rstp-simple-server using the RTSP protocol (in this case using TCP for simplicity).

You can check the stream with VLC (or any other client) reading from "rtsp://127.0.0.1:8554/test/stream". And you can check that it works like a charm. In fact, you can stream any SPTS with multiple audio pids, teletext streams, and other data info. And it supports any regular codec that the MPEG-TS container can handle. So it's a very interesting solution for different scenarios.

The problem, however is when you want to streaming this services to HLS clients. In this case the typical message (the stream doesn't contain an H264 track or an AAC track) appears in the LOG. Does this mean that it is not possible to stream this program in HLS? The technical answer is NO. You can convert mainly any MPEG-TS to an HLS. However, what we need to achieve it with rtsp-simple-server?

Here's my suggestion to enhance this functionality after a small review of the code:

• What the HLS muxer does at time it's more or less: Capture all RTP packets from audio and video, decode them, and remux them (without reencoding). The limitations are that video could be only H264 and audio AAC.
• However, a third stream could be supported: the encapsulated mpegts track. Internally this will not be complex, as it only requires some identification of the stream.
• After adding the support of encapsulated mpegts tracks, the HLS segmenter/muxer process could be modified to support this type of tracks. In this case the most simple strategy to create the chunks could be this: store PAT and PMT pids in a internal buffer, and when arrive a new video track with the PUSI flag set then the start of a new chunk. In this chunk you copy first the last received PAT and PMT pids, and then all the new coming pids until a new PUSI video packet arrives. This will not be a perfect HLS segmentation, but it will work. Other enhancements could be done after to enhance the provided functionality.

What you think about this idea? I hope someone will want to work on this. Or implement a solution to use an external segmenter (another technical solution). Regards.

[aler9]

Hello, of course it would be good to support RTSP streams with a single track that contains a MPEG-TS stream that multiplexes multiple tracks, but... in my opinion using RTSP with MPEG-TS is really a waste of bandwidth.

RTSP already supports natively multiple tracks and supports much more codecs than MPEG-TS; furthermore, it is way more efficient since the track list is sent once, there are no PAT, PMTs and no padding - these features are needed by MPEG-TS since it's a stream format designed for broadcast and there is no handshake.

Regarding the HLS segmentation, it is currently performed by looking at H264 NALUs and by finding IDR NALUs - this works with any stream container. Adding support for MPEG-TS would simply mean extracting H264 NALUs from MPEG-TS, then the existing algorithm will work regardless of the original stream format.

[lars18th]

Hi @aler9 ,

Thank you for your response! I'm glad to discuss this topic with you.

First of all, RTSP with MPEG-TS have sense in several scenarios. For example if you want to pass the PSI tables in the stream for a DVB environment. IMHO the support of MPEG-TS encapsulation as an option is valuable. Then the user can select to use it or not. More or less is the option to have all tracks encapsulated in one track (then the track 0 is "data", instead of "video"). So, please don't remove this support and continue considering this escenario.

Futhermore, perhaps it has sense to enhance a bit the current support. For example, you could add support to identify a track as "data", and then if it's only one track in the live stream, then this implies that the stream is encapsulated. This information could be then used to do a demuxing process (done with an external tool) if a subtrack is necessary. I'm not saying to support this directly and now. Only to have the option to check and trace it. You agree with that? You feel it has sense?

Regarding with the HLS sementation I'm not sure about your proposal. You say to "copy" the track 0 stream (the MPEG-TS) to ffmpeg to extract the H.264 PES, process this with the current algoritm, and then use this information to segment the original track 0 stream? You think this will work?

Why not adopt another strategy and support external segmenters? For example if we can pass the MPEG-TS stream to ffmpeg, it could create the M3U8 file (or the ffconcat file shared over a PIPE) and store the chunks in the disk, and use them internally. This could work?

Please, comment about this. I want to know your feeling. Regards.

[aler9]

First of all, RTSP with MPEG-TS have sense in several scenarios. For example if you want to pass the PSI tables in the stream for a DVB environment

If MPEG-TS carries more metadata, of course we can add support for it, and in particular we can support passing PSI tables to MPEG-TS segments, but keep also in mind that in the future the server will switch to fMP4 segments and drop MPEG-TS segments, in order to support low latency HLS, that requires this change.

You think this will work?

The problem is that we must support streams that come from various protocols, and the only way to do this is using an universal algorithm.

Why not adopt another strategy and support external segmenters?

Because the aim of this project is creating a lightweight and ready-to-use executable that replaces a lot of oversized software.

[Curid]

The problem, however is when you want to streaming this services to HLS clients.

Why not send it directly over HLS or DASH? What's the point of using a RTSP server if you don't need sub-second latency?

[hhackbarth]

Unfortunately, audio and video are in some cases not in sync, when I send them as multiple streams over RTSP. When I consume it with mpv, I see differences of up to 5 seconds.

When I consume it with a browser using HLS, there is no audio at all, I see only the video.

Test-Pipeline: gst-launch-1.0 -e videotestsrc is-live=true ! video/x-raw, width=1280, height=720, framerate=30/1, format=NV12 ! queue ! videoconvert ! queue ! x264enc bitrate=2500 ! s.sink_0 audiotestsrc ! queue ! audioconvert ! queue ! avenc_aac bitrate=128000 ! s.sink_1 rtspclientsink name=s location=rtsp://localhost:8554/live latency=1000 protocols=tcp+udp-mcast+udp rtx-time=0 With this pipeline, I could not reproduce the audio/video desynchronisation but at least it shows, that HLS has no audio.

When I mux audio and video in MPEGTS and pack this to RTSP, audio and video are always in sync, when consuming the streams with mpv.

Example pipeline: gst-launch-1.0 -e videotestsrc is-live=true ! video/x-raw, width=1280, height=720, framerate=30/1, format=NV12 ! queue ! videoconvert ! queue ! x264enc bitrate=2500 ! mux. audiotestsrc ! queue ! audioconvert ! queue ! avenc_aac bitrate=128000 ! mux. mpegtsmux name=mux alignment=7 ! rtspclientsink location=rtsp://localhost:8554/live latency=1000 protocols=tcp+udp-mcast+udp rtx-time=0

This can be consumed e.g. with mpv or vlc but when I try to request it as HLS in a browser on port 8888, I get only 404 not found.

So, my whish would be to have always audio and video in sync (using the MPEGTS muxing) and to have it both also using HLS in a browser.

If you are working on low-latency-HLS, perhaps my whishes could come true there?

[aler9]

Ingesting MPEG-TS streams is supported since some versions, with both UDP and SRT, and audio+video synchronization is preserved since PTS is copied from MPEG-TS to HLS, so this issue is fixed.

[aler9]

Just a note: remuxing MPEG-TS from RTSP to HLS is still unsupported, but you can send MPEG-TS with UDP or SRT, which is way more efficient.

[github-actions[bot]]

This issue is being locked automatically because it has been closed for more than 6 months. Please open a new issue in case you encounter a similar problem.