Receiving and playing a MPEG-TS UDP (Legacy) stream from VLC

[torin-eviation]

Hi there, I was hoping to get some advice on how to record and play an MPEG-TS stream received over UDP.

So far, I've mostly followed the example in the Mp3StreamingDemo. I think the breakdown in my understanding is when, in the demo, they make an HTTP request and are calling GetResponseStream() to get a stream for loading frames. In my implementation, I'm receiving raw UDP packets and constructing a new MemoryStream with the bytes each time, then loading frames from those. I get about half a second of garbled noise, and then silence. Here's my code so far:

    internal class UdpAudioReceiver
    {
        // UDP socket
        private UdpClient _socket;
        private IPEndPoint _udpEndPoint = new(0, 0);
        private readonly UdpState _udpState;

        // MP3 decompressor
        private Mp3WaveFormat? _waveFormat;
        private DmoMp3FrameDecompressor? _decompressor;

        // Provider/player
        private BufferedWaveProvider? _waveProvider;
        private WaveOut? _player = new();

        private struct UdpState
        {
            public UdpClient UdpClient;
            public IPEndPoint IpEndPoint;
        }

        public UdpAudioReceiver(int port)
        {
            this._socket = new(1234, AddressFamily.InterNetwork);
            this._udpState = new UdpState
            {
                IpEndPoint = this._udpEndPoint,
                UdpClient = this._socket
            };
        }

        public void Start()
        {
            this._socket.BeginReceive(this.ReceiveCallback, this._udpState);
        }

        private void ReceiveCallback(IAsyncResult ar)
        {
            var u = ((UdpState)ar.AsyncState).UdpClient;
            var e = ((UdpState)ar.AsyncState).IpEndPoint;

            var receivedBytes = u.EndReceive(ar, ref e);

            var stream = new MemoryStream(receivedBytes);

            Mp3Frame frame;
            try
            {
                frame = Mp3Frame.LoadFromStream(stream);
            }
            catch (Exception)
            {
                this._socket.BeginReceive(this.ReceiveCallback, this._udpState);
                return;
            }

            // Create format and decompressor if we need to
            this._waveFormat ??= new(frame.SampleRate, frame.ChannelMode == ChannelMode.Mono ? 1 : 2, frame.FrameLength,
                frame.BitRate);
            if (this._decompressor == null)
            {
                this._decompressor = new DmoMp3FrameDecompressor(this._waveFormat);
                this._waveProvider = new(this._decompressor.OutputFormat);
                this._waveProvider.BufferDuration = TimeSpan.FromSeconds(1);
                this._waveProvider.DiscardOnBufferOverflow = true;

                this._player = new();
                this._player.Init(this._waveProvider);
            }

            // Decompress frame
            var decompressed = new byte[16384 * 4];
            int decompressedLength;
            try
            {
                decompressedLength = this._decompressor.DecompressFrame(frame, decompressed, 0);
            }
            catch
            {
                decompressedLength = 0;
            }

            if (decompressedLength > 0)
            {
                this._waveProvider!.AddSamples(decompressed, 0, decompressedLength);
            }

            if (this._waveProvider!.BufferedBytes == this._waveProvider.BufferLength)
            {
                this._player!.Play();
            }

            this._socket.BeginReceive(this.ReceiveCallback, this._udpState);
        }

[Zintom]

I'm not familiar with UDP sockets, however, I do know that with TCP sockets you need to ensure that all your data is received when calling EndReceive (sometimes calling EndReceive multiple times to get all the data from the winsoc API).

Are you ensuring the integrity of your received data? Eliminating this as a factor will help you isolate whether the problem is down to your code, or NAudio.

[Zintom]

I've done a bit more thinking about this. May I ask why you are using UDP over TCP? With UDP you can't guarantee that all your bytes will arrive, An incomplete UDP packet could break playback. Surely it makes much more sense to use TCP?

Also, have you tried your code with a mock implementation of the UDP server? that way you could eliminate your network code being the issue all together.

Kind regards, Tom

[torin-eviation]

Hey Tom, thanks for your response. To answer the question as to why UDP: our network topology in this use case involves streaming audio over a one-way RF link to a receiving machine. Because it's only half-duplex, that eliminates the possibility of any protocols that require a handshake like TCP. The network is also lossy and prone to dropping/chopping packets, which breaks playback in VLC, our initial solution. I was hoping that the frame-level control we get with NAudio would let us detect such losses and resume playback when the network stabilizes again.

With the issue I described in my initial comment, I've been streaming UDP over the loopback interface and receiving it with UdpAudioReceiver above. I can watch new packets being received and decompressed, so I don't believe the network code to be the issue.

[Zintom]

Hi Torin,

Have you considered sending a hash or CRC alongside each frame you send? That way you could verify that the data received is valid before passing it to the Mp3Frame object. Any time invalid data was received you could just fill the gap in time with silent audio so that the playback wouldn't stop or glitch.

This is just a random idea, I'm no expert in the library and for all I know there may be a way to do this automatically in NAudio.

Tom.

[torinv]

That's a great idea for data validation once we get this onto our actual network. But for the time being, I'm using my local machine's loopback interface, and I'm not able to get playback working. I was hoping to get some help with raw UDP playback on a stable network.

[markheath]

Sorry for the late reply. With apps like this I always recommend first writing code that writes the received (decompressed) audio to a WAV file. Then you can play that WAV file and check your decoding side of the app is working as expected without glitches or garbled audio. Once that's the case, you can then focus on the realtime playback.

One issue I noticed with your code is that you are waiting until the BufferedWaveProvider is full before starting to play. That could result in it needing to throw away audio since other packets might be added. I'd recommend using a longer duration for the BufferedWaveProvider (say 10 seconds), and then starting playback once there is at least 1 second's worth of audio buffered.