Avoid audible glitch when clipping and re-concatenating a single audio source

[Yahui-Li]

It works perfect when play a single .m4a file with ProgressiveMediaSource. But it seems not sealmess when play the same audio file with ClippingMediaSource and ConcatenatingMediaSource. For example

        val durations = longArrayOf(20100, 467, 11233, 7100, 6267, 10300)
        var start = 0L
        val audioFile = File(dir, "sample.m4a")
        val audioSource = DefaultMediaSourceFactory(context)
            .createMediaSource(MediaItem.fromUri(Uri.fromFile(audioFile)))
        val concatAudioSource = ConcatenatingMediaSource(true)
        for (i in 0 until durations.size) {
            val durationUs = durations[i] * 1000
            val end = start + durationUs
            val clipAudio = ClippingMediaSource(audioSource, start, end)
            concatAudioSource.addMediaSource(clipAudio)
            start = end
        }
        _player?.run {
            setMediaSource(concatAudioSource)
            prepare()
        }

then you will heard that an obvious pause(maybe glitch?, I'm not sure) between two period. It's easy to reproduce

[icbaker]

I've reproduced this in the demo app and can hear a playback glitch. I used the following JSON (which gets coverted to a playlist of 2 MediaItems, and this ends up being handled very similarly to ClippingMediaSource and ConcatenatingMediaSource):

{
  "name": "audio playlist",
  "playlist": [
    {
      "uri": "path to mp3",
      "clip_start_position_ms": 150000,
      "clip_end_position_ms": 160000
    },
    {
      "uri": "path to mp3",
      "clip_start_position_ms": 160000,
      "clip_end_position_ms": 170000
    }
  ]
},

@krocard can you take a look at what's happening here?

[Yahui-Li]

Any progress?

[krocard]

This is a relatively contrived use case. Clipping media source doesn't cut a resource at an exact pcm frame, which perfect gapless playback would need. ExoPlayer supports gapless playback, but the amount to clip needs to be small and present in the container, not the media source.

In general a player that would not introduce a ramp down and up on transitions would result in an audible glitch as the waveform would not be continuous (high frequency). Thus the current behaviour is desirable.

Could you tell us more about what you are trying to achieve?

[Yahui-Li]

See, I want to merge a sequence of video resources and a single audio resource. For example, I have three video fragments each duration is 10 sec and an audio fragment that duration is 30 sec. Now I want to merge those resources together but I can't because the ConcatenatingMediaSource of videos had different period count with the audio MediaSource, so I cut the audio to three pieces then concatenate them so that I can merge the video and audio together. I can't find another way to handle this situation. Do you have any suggestion?

[krocard]

You should use a MergingMediaSource to merge

• the audio source
• a ConcatenatingMediaSource of the video sources.

[krocard]

@marcbaechinger Can you confirm if this is correct? MergingMediaSource has a restriction that each source should have the same number of periods, so I'm not sure it's compatible with ConcatenatingMediaSource.

[marcbaechinger]

Yes, I can confirm that MergingMediaSource.createPeriod() expects that all sources have the same number of periods.

[Yahui-Li]

So, What's the most appropriate way to solve this? I really need help.

[krocard]

Thanks for the confirmation @marcbaechinger. So it seems my suggestion to concatenate the videos and then merge them with the audio will not work due to the difference in period. @marcbaechinger, given this limitation, what would be the best way to play multiple videos with continuous audio?

[marcbaechinger]

I don't think that there is currently a solution for this with ExoPlayer media source composition for the reasons given above. I marked this issue as an enhancement. We need to look into this and see how we can provide a solution for this. We see several approaches how we can make this work, but I can't give you a concrete timeline for when that will happen I'm afraid.

An option to have this working properly in the mean time, would be to merge the media before you play it. You could for instance create a single video stream with the audio and the three video parts with ffmpeg or a similar tool and then play the video as a single media file with ExoPlayer. I understand that this is not a dynamic solution that you can apply easily on-the-fly with arbitrary videos, but if this use case is specific to these four media files this is probably the easiest way.

Sorry, for not being able to give you a better answer.

[yogeshchoudhary-gts]

+1

[Yahui-Li]

And I tried that if I cut a resource at an exact pcm frame(I use atds as audio format), there will produce a glitch either. so I guess the reason why produce a glitch is not because ExoPlayer can not cut audio resource at an exact pcm frame but there has no pre-cache of next period when the active period is playing. But I'm not sure about that. I will read source code more deeply when I have time。

[tonihei]

the reason why produce a glitch

The reason is that ExoPlayer flushes the MediaCodec between the different audio resources. If you disable the flushing manually and keep the same codec instance, everything works smoothly without any audible glitches.

We have to flush the codec in the general case because we can't know whether feeding data from two audio sources to the same decoder consecutively is save (=doesn't result in some other form of audible glitch). So this enhancement is tracking to either:

• keep the decoder in some cases where we can be sure it's safe (unclear if there is such a case)
• signal to the decoder that two consecutive audio sources are continuous and can use the same decoder. This only makes sense in combination with ClippingMediaSource, so needed to be forwarded from this source somehow.
• Support merging sources with different number of periods. This is the most flexible approach as it avoids clipping to start with. It doesn't fit very well into our media model though.

[Yahui-Li]

Wow，thank you for your timely and clearly response. I'm appreciate it very much and it's very useful to me. I try to reuse the codec and It's seems perfect! Thanks, my friend.

[ShahyarTaheri]

@YouheiLee @tonihei Would you be able to provide a sample on how to manually disable codec flushing? I have been struggling with this issue for a while and could not find a solution. Thank you

[tonihei]

I hackily changed this line to return REUSE_RESULT_YES_WITHOUT_RECONFIGURATION. This is only safe for this particular use case though and no other audio transition as pointed out above.