2.0 alpha1 (docker): testing video broadcasting with fallback

[xorinzor]

I've started playing around with 2.0.0-alpha1 a bit using the docker image (liquidsoap:2.0.0-alpha1), but keep running into this message.

[decoder:3] Available decoders cannot decode "/home/liquidsoap/app/resources/fallback.mp4" as {audio=pcm(stereo),video=yuva420p,midi=none}

How can I properly transcode all input to the output that I want, regardless of it's type?

I've been trying quite a few different things. also using decode instead of encode for the ffmpeg methods. But with the same result unfortunately.

What I'm effectively trying out is to have a video fallback source (which also has a silent audio track). Then the playlist can contain either audio-only, or audio+video files. If an audio-only file is played, it should use the video track from the fallback source.

The output should be of a format that I specify, anything else should be transcoded.

This is the bit of test code I've been playing around with:

output_format = %ffmpeg(
  format="webm", 
  %audio(
    codec="aac",
    b="128k",
  ),
  %video(
    codec="libx264",
    b="5M"
  )
)

#Create our unfallible source
silent_source = single("/home/liquidsoap/app/resources/fallback.mp4")
#silent_source = ffmpeg.encode.audio_video(output_format, silent_source)

#Requester input
playlist_source = playlist("/home/liquidsoap/app/playlist.txt",id="queue_list",reload_mode="seconds",reload=10,timeout=5.)
#playlist_source = ffmpeg.encode.audio_video(output_format, playlist_source)

# Add the fallback silent unfallible-source and smart crossfade
live = fallback(track_sensitive=false, [playlist_source, silent_source])

#live = ffmpeg.encode.audio_video(output_format, live)

#stream output
output.srt(
  id = "test_output",
  mode = "listener",
  host = "localhost",
  port = 8000,
  bind_address = "0.0.0.0",
  output_format,
  fallible = true,
  live
)

[toots]

Thanks for this report! Could you email me a copy of the file at [email protected]? Thks!

[xorinzor]

Sent!

[toots]

Got it, thanks! It looks like the file does not contain any audio:

% ffprobe fallback.mp4
(...)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'fallback.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.56.101
  Duration: 00:00:10.02, start: 0.000000, bitrate: 1494 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 1488 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    Metadata:
      handler_name    : VideoHandler

[xorinzor]

Ah, yes, it seems that's the issue! Didn't think to check the video file, the message didn't really seem to indicate anything was wrong with the file either.

Either way, adding a silent track does seem to fix this message. Thanks!

[xorinzor]

Looks like it still has some issues. It doesn't seem to fallback to the original video track when an mp3 is played from the playlist The message is the same as in the original issue (except this time for the mp3 file), which is to be expected since it apparently expects a file with both an audio and video stream.

How should I go about handling this? Given that playlist_source can contain both audio-only and audio+video files.

[toots]

The system is designed to only accept files that have the right content. If the playlist has a mix of audio only and audio+video files, only the audio+video files should be picked.

What you describe here sounds like a bug. Would you be able to share some logs of the case where a mp3 file is played?

[xorinzor]

It's not actually playing mp3 files, it's throwing the same error message, probably because it's expecting a video track to be present.

Is it possible to implement some kind of has_video method?

That way I could just create a separate audio and video source using drop_audio and drop_video and merge them back together before outputting them.

Could achieve the same by using 2 separate playlists (one for audio, and one for video), but that'd be a lot more difficult to work around with queueing.

[toots]

I see. So, ideally you would like the fallback to complete whichever part is missing in the files from the playlist?

[xorinzor]

exactly!

The fallback video is the reliable source with both an audio and video track.

Then the playlist will contain either requested songs, or videos.

If a song is playing, it should use the video track from the fallback video.

If a video is playing, it can use both the audio and video from that source.

Of course I'll be adding some bits around it, but that's the basic idea :)

[xorinzor]

Thinking about it, a has_video method would in my case be very helpful, not just for this. But also for the other ideas that I had in regards to post-processing.

From what I could tell reading through all the documentation there isn't really such a method available yet

[toots]

I'll think about it. It's a pretty big departure from our type and runtime model. We assign each stream a fixed content before starting the streaming loop. One thing I could think about is a optional content dropping or merging, something like add_audio_if_source_does_not_have_some. I'll see about adding it but for now the focus is on fixing issues.

[xorinzor]

Hm, I can understand that might require some changes.

If you do plan on adding such a method, might I suggest doing add_video_if_source_does_not_have_some instead? I think having a video without audio track is less common then an audio file without video track.

I'll see if I can come up with a workaround in the meanwhile.

[toots]

Yes, I would of course add it both ways.

[xorinzor]

@toots do you have any update as for when this might be implemented?

[toots]

Hey! I'm not sure yet, sorry. We're restarting some fundamental work on generalizing content to be more flexible with content, allow subtitles, multi track audio and etc. This feature should hopefully be easier to tackle after that.

[toots]

This is done now! Check this out: https://www.liquidsoap.info/doc-dev/multitrack.html#add-a-default-video-track