Merge audio with optional video

[brahmkshatriya]

I am trying to merge audio and video sources seperately using MergingMediaSource(), my problem is that I can only know if my mediaItem has a video or not when the mediaItem has been resolved, so I cannot create a MediaSource after the mediaItem has been resolved. Which lead me to think, since I cannot know if i will have just audio or both audio and video, i should always create 2 mediaSources, if the video doesnt exist , then I can just make it send empty data.

So my question is, is there a way to send empty data to via the DataSourceFactory. I thought it could be done via doing this

@OptIn(UnstableApi::class)
class VideoDataSource(val factory: DefaultDataSource.Factory) : BaseDataSource(true) {

    class Factory(context: Context) : DataSource.Factory {
        private val defaultDataSourceFactory = DefaultDataSource.Factory(context)
        override fun createDataSource() = VideoDataSource(defaultDataSourceFactory)
    }

    private var source: DataSource? = null

    override fun getUri() = source?.uri ?: "".toUri()

    override fun read(buffer: ByteArray, offset: Int, length: Int) =
        source?.read(buffer, offset, length) ?: RESULT_END_OF_INPUT

    override fun close() {
        source?.close()
        source = null
    }
    
    override fun open(dataSpec: DataSpec): Long {
        val video = dataSpec.customData as? StreamableVideo ?: return -1
        val spec = video.request.run {
            dataSpec.copy(uri = url.toUri(), httpRequestHeaders = headers)
        } 
        val source = factory.createDataSource()
        this.source = source
        return source.open(spec)
    }
}

But when using this, it gives me this exception UnrecognizedInputFormatException: None of the available extractors (FlvExtractor, FlacExtractor, WavExtractor, FragmentedMp4Extractor, Mp4Extractor, AmrExtractor, PsExtractor, OggExtractor, TsExtractor, MatroskaExtractor, AdtsExtractor, Ac3Extractor, Ac4Extractor, Mp3Extractor, AviExtractor, JpegExtractor, PngExtractor, WebpExtractor, BmpExtractor, HeifExtractor) could read the stream.{contentIsMalformed=false, dataType=1}

This is my CustomMediaSourceFactory

@OptIn(UnstableApi::class)
class CustomMediaSourceFactory(
    val context: Context,
) : MediaSource.Factory {

    private val audioSource = DefaultMediaSourceFactory(context)
    private val videoSource = DefaultMediaSourceFactory(context)

    override fun setDrmSessionManagerProvider(
        drmSessionManagerProvider: DrmSessionManagerProvider
    ): MediaSource.Factory {
        audioSource.setDrmSessionManagerProvider(drmSessionManagerProvider)
        videoSource.setDrmSessionManagerProvider(drmSessionManagerProvider)
        return this
    }

    override fun setLoadErrorHandlingPolicy(
        loadErrorHandlingPolicy: LoadErrorHandlingPolicy
    ): MediaSource.Factory {
        audioSource.setLoadErrorHandlingPolicy(loadErrorHandlingPolicy)
        videoSource.setLoadErrorHandlingPolicy(loadErrorHandlingPolicy)
        return this
    }

    fun setSourceFactory(
        audioFactory: DataSource.Factory,
        videoFactory: DataSource.Factory
    ): CustomMediaSourceFactory {
        audioSource.setDataSourceFactory(audioFactory)
        videoSource.setDataSourceFactory(videoFactory)
        return this
    }

    override fun getSupportedTypes() = videoSource.supportedTypes

    override fun createMediaSource(mediaItem: MediaItem): MediaSource {
        // I do not know if mediaItem will have video streams or not, yet.
        // Only known, once the mediaItem is loaded/ resolved by the ResolvingDatasSource
        return MergingMediaSource(
            true,
            false,
            audioSource.createMediaSource(mediaItem),
            videoSource.createMediaSource(mediaItem),
        )
    }
}

So how do I approach this? is there a way to send empty data from a MediaSource? or is there a better way to approach this?

[icbaker]

But when using this, it gives me this exception UnrecognizedInputFormatException: None of the available extractors (FlvExtractor, FlacExtractor, WavExtractor, FragmentedMp4Extractor, Mp4Extractor, AmrExtractor, PsExtractor, OggExtractor, TsExtractor, MatroskaExtractor, AdtsExtractor, Ac3Extractor, Ac4Extractor, Mp3Extractor, AviExtractor, JpegExtractor, PngExtractor, WebpExtractor, BmpExtractor, HeifExtractor) could read the stream.{contentIsMalformed=false, dataType=1}

This sounds expected to me. You effectively asked the library to try and work out what type of file an empty byte array represents, and it couldn't. How could it? There's no signal in empty bytes...

I don't think heading down the path of a DataSource that returns zero bytes is the right way to approach your problem.

I think what you want is a MediaSource that doesn't publish any tracks (i.e. work at a different level of abstraction), and then merge that with one that does.

@tonihei might have some more thoughts.

[tonihei]

I think what you want is a MediaSource that doesn't publish any tracks (i.e. work at a different level of abstraction), and then merge that with one that does.

There seem to be two different problems here if I understand the requirement correctly.

1. If source A has both audio and video, source B should ideally not even exist or not publish any tracks at all. This could be done as a MediaSource that publishes an empty list of tracks, but if generating the source with no data is cheap, I wouldn't even bother and just always create the placeholder source B. Then you can instruct the track selection process to always prefer the first one in the list if that's not already happening by default.
2. If source A has only audio, source B needs to generate an empty list of video samples. I'm not aware of a utility to do this easily at the moment. You could probably copy SingleSampleMediaSource/MediaPeriod and remove the single sample to make it a NoSampleMediaSource. As @icbaker already highlighted, creating a zero byte data source won't work unfortunately.

Taking a step back though - what do you need this empty video track for exactly? You said something about trying to merge audio and video sources separately, so I'm wondering if there is a completely different approach to your actual problem.

[brahmkshatriya]

Taking a step back though - what do you need this empty video track for exactly? You said something about trying to merge audio and video sources separately, so I'm wondering if there is a completely different approach to your actual problem.

This is what I want to do

I am successful to do it, if the audio uri only contains audio data and if video exists and only contains video data, Ideally, I would like to create a MediaSource that handles to merge both audio from audio uri and video from video uri

Currently I am passing a custom data class to contain the audio and video uri inside the resolved dataspec, which is then used by both audio source and the video source.

I think what you want is a MediaSource that doesn't publish any tracks (i.e. work at a different level of abstraction), and then merge that with one that does.

Yes, I think that would be the appropriate solution, instead of just sending an empty datasource

1. If source A has both audio and video, source B should ideally not even exist or not publish any tracks at all. This could be done as a MediaSource that publishes an empty list of tracks, but if generating the source with no data is cheap, I wouldn't even bother and just always create the placeholder source B. Then you can instruct the track selection process to always prefer the first one in the list if that's not already happening by default.
2. If source A has only audio, source B needs to generate an empty list of video samples. I'm not aware of a utility to do this easily at the moment. You could probably copy SingleSampleMediaSource/MediaPeriod and remove the single sample to make it a NoSampleMediaSource. As @icbaker already highlighted, creating a zero byte data source won't work unfortunately.

I looked through the SingleSampleMediaSource & SingleSampleMediaPeriod classes, I would like to when they receive the resolved dataspec? but I guess this is not the ideal solution for me either, since it does not ignore the audio from video source or video from the audio source

[tonihei]

Thanks for the drawing!

Ignoring the DataSpec resolution for a second, the merging logic can probably be done by:

new MergingMediaSource(
   new FilteringMediaSource(audioSource, C.TRACK_TYPE_AUDIO),
   new FilteringMediaSource(audioSource, C.TRACK_TYPE_VIDEO))

The FilteringMediaSource makes sure to only publish tracks of the given type, ignoring all other tracks if they exist.

For the part about resolving the URL - how dynamic is this resolution? That is, does it have to happen immediately before playback, or is the result always the same anyway?

• If the resolution is static, you could just resolve the Uri outside of ExoPlayer and then either add the MergingMediaSource as above, or just the audioSource as needed.
• If you prefer to have the logic inside the player flow (e.g. to make sure the UI already shows this item as buffering, or because it has to happen just-in-time), then I guess you need to write a subclass of CompositeMediaSource<Void> that resolves the URL first and then creates the wrapped source:

private static final class DelayedSource extends CompositeMediaSource<Void> {

  private final MediaItem mediaItem;
  private MediaSource actualSource;

  public DelayedSource(MediaItem mediaItem) {
    this.mediaItem = mediaItem;
  }

  @Override
  protected void prepareSourceInternal(@Nullable TransferListener mediaTransferListener) {
    // Start URL resolution (needs a background thread, e.g. using Loader class)
  }

  private void onUrlResolved(Uri audioUrl, @Nullable Uri videoUrl) {
    actualSource = /* create MergingMediaSource or just audio source here */;
    prepareChildSource(null, actualSource);
  }

  @Override
  protected void onChildSourceInfoRefreshed(Void childSourceId, MediaSource mediaSource,
      Timeline newTimeline) {
    refreshSourceInfo(newTimeline);
  }

  @Override
  public MediaItem getMediaItem() {
    return mediaItem;
  }

  @Override
  public MediaPeriod createPeriod(MediaPeriodId id, Allocator allocator, long startPositionUs) {
    return actualSource.createPeriod(id, allocator, startPositionUs);
  }

  @Override
  public void releasePeriod(MediaPeriod mediaPeriod) {
    return actualSource.createPeriod(mediaPeriod);
  }
}

If that is a useful utility class, we can also consider adding it to the library.

[brahmkshatriya]

Thank you @tonihei , this was very helpful to know!

What If new metadata of the mediaItem is resolved, I cant just use player.replaceMediaItem() here, because it will cause the player reload the mediaSource infinitely, so where should I notify the player that media item has been updated, since it does not automatically detect it.

[brahmkshatriya]

@tonihei turns out it does change the mediaItem, if it changes data, but now once in a while, I get this error

java.lang.IllegalStateException
    at androidx.media3.common.util.Assertions.checkStateNotNull(Assertions.java:117)
    at androidx.media3.exoplayer.source.BaseMediaSource.getPlayerId(BaseMediaSource.java:185)
    at androidx.media3.exoplayer.source.CompositeMediaSource.prepareChildSource(CompositeMediaSource.java:122)
    at example.DelayedSource.access$prepareChildSource(DelayedSource.kt:43)
    at example.DelayedSource$onUrlResolved$2.invokeSuspend(DelayedSource.kt:81)

I dont exactly know what causes it, If it helps this is how I am using the DelayedSource

@OptIn(UnstableApi::class)
class DelayedSource(
    private val mediaItem: MediaItem,
    private val scope: CoroutineScope,
    private val audioFactory: MediaFactories,
    private val videoFactory: MediaFactories,
) : CompositeMediaSource<Nothing>() {

    private var resolvedMediaItem: MediaItem? = null
    private lateinit var actualSource: MediaSource

    override fun prepareSourceInternal(mediaTransferListener: TransferListener?) {
        super.prepareSourceInternal(mediaTransferListener)
        scope.launch(Dispatchers.IO) {
            val new = resolve(mediaItem)
            onUrlResolved(new)
        }
    }

    private suspend fun onUrlResolved(new: MediaItem) = withContext(Dispatchers.Main) {
        resolvedMediaItem = new
        val video = new.video
        val source = when (val video = new.video) {
            null -> null
            is Streamable.Media.WithVideo.WithAudio -> videoFactory.create(new)
            is Streamable.Media.WithVideo.Only -> if (!video.looping) MergingMediaSource(
                FilteringMediaSource(videoFactory.create(new), C.TRACK_TYPE_VIDEO),
                FilteringMediaSource(audioFactory.create(new), C.TRACK_TYPE_AUDIO)
            ) else null
        }
        actualSource = source ?: FilteringMediaSource(audioFactory.create(new), C.TRACK_TYPE_AUDIO)
        prepareChildSource(null, actualSource)
    }

    override fun getMediaItem() = resolvedMediaItem ?: mediaItem

    override fun createPeriod(
        id: MediaSource.MediaPeriodId, allocator: Allocator, startPositionUs: Long
    ) = actualSource.createPeriod(id, allocator, startPositionUs)

    override fun releasePeriod(mediaPeriod: MediaPeriod) =
        actualSource.releasePeriod(mediaPeriod)

    override fun onChildSourceInfoRefreshed(
        childSourceId: Nothing?, mediaSource: MediaSource, newTimeline: Timeline
    ) = refreshSourceInfo(newTimeline)
    
    private suspend fun resolve(mediaItem: MediaItem): MediaItem {
        //...
    }
}

Edit: You said to use a Loader Class, I dont exactly know what you meant by that, but I implemented this using a kotlin coroutine scope

[brahmkshatriya]

Also doing something like this, didnt use to cause the mediaItem to be loaded again

val newItem = item.run {
    buildUpon().setMediaMetadata(
        mediaMetadata.buildUpon().setUserRating(ThumbRating(liked)).build()
    )
}.build()
player.replaceMediaItem(session.player.currentMediaItemIndex, newItem)

But after shifting to the DelayedSource, it reloads the whole thing again, how can I prevent it?

Edit : I fixed it by adding the following lines to the DelayedSource

    override fun canUpdateMediaItem(mediaItem: MediaItem) =
        actualSource.canUpdateMediaItem(mediaItem)

    override fun updateMediaItem(mediaItem: MediaItem) =
        actualSource.updateMediaItem(mediaItem)

[brahmkshatriya]

androidx.media3.exoplayer.ExoPlaybackException: Unexpected runtime error
 at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:720)
 at android.os.Handler.dispatchMessage(Handler.java:102)
 at android.os.Looper.loopOnce(Looper.java:257)
 at android.os.Looper.loop(Looper.java:368)
 at android.os.HandlerThread.run(HandlerThread.java:67)
Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'void androidx.media3.exoplayer.hls.playlist.DefaultHlsPlaylistTracker$MediaPlaylistBundle.maybeThrowPlaylistRefreshError()' on a null object reference
at androidx.media3.exoplayer.hls.playlist.DefaultHlsPlaylistTracker.maybeThrowPlaylistRefreshError(DefaultHlsPlaylistTracker.java:225)
 at androidx.media3.exoplayer.hls.playlist.DefaultHlsPlaylistTracker.maybeThrowPrimaryPlaylistRefreshError(DefaultHlsPlaylistTracker.java:219)
 at androidx.media3.exoplayer.hls.HlsMediaSource.maybeThrowSourceInfoRefreshError(HlsMediaSource.java:510)
 at androidx.media3.exoplayer.source.CompositeMediaSource.maybeThrowSourceInfoRefreshError(CompositeMediaSource.java:61)
 at androidx.media3.exoplayer.source.CompositeMediaSource.maybeThrowSourceInfoRefreshError(CompositeMediaSource.java:61)
 at androidx.media3.exoplayer.source.MergingMediaSource.maybeThrowSourceInfoRefreshError(MergingMediaSource.java:198)
 at androidx.media3.exoplayer.source.CompositeMediaSource.maybeThrowSourceInfoRefreshError(CompositeMediaSource.java:61)

Also since I switched, I keep getting this error, from time to time

[tonihei]

but now once in a while, I get this error

I think this might be at threading issue. The code above uses Dispatchers.IO for onUrlResolved, but methods like prepareChildSource should be called on the same thread that originally called prepareSourceInternal. This is likely also the cause of the second exception (the NullPointerException).

[brahmkshatriya]

but now once in a while, I get this error

I think this might be at threading issue. The code above uses Dispatchers.IO for onUrlResolved, but methods like prepareChildSource should be called on the same thread that originally called prepareSourceInternal. This is likely also the cause of the second exception (the NullPointerException).

but now once in a while, I get this error

I think this might be at threading issue. The code above uses Dispatchers.IO for onUrlResolved, but methods like prepareChildSource should be called on the same thread that originally called prepareSourceInternal. This is likely also the cause of the second exception (the NullPointerException).

Then what should I use instead?

[tonihei]

Then what should I use instead?

You can use Dispatchers.IO for the background work. However, once the work is done onUrlResolved should not be called with Dispatchers.Main but with a dispatcher for the Looper thread that called prepareSourceInternal. I think you can that by calling Handler(Looper.myLooper).asCoroutineDispatcher() within prepareSourceInternal. This is more of a generic Kotlin question though and I'm no expert in that.