Garbled output and enocoding can't keep up with output with multiple ffmpeg filters (with sporadic seg faults)

[Russsgithub]

Describe the bug I've been trying to make video from an audio stream which adds now playing text, cover image and a waveform extracted from the audio (using ffmpeg filter showwaves).

I can make it work well without showwaves, and with just showwaves, but if i add the two together I get garbled output ( on both ogg and mp3 streams ( ogg is my target ) ), which sounds like an accelerated buffer trying to keep up and a stream that can't keep up with the output.

I get sporadic segmentation faults using this code, mostly it'll run and sometimes not. With no other information.

CPU usage is nothing too dramatic on an old 3rd gen i7 ( 2 processes are running at 85% and 55% and another collection are running at from 0% to 5% ), ram is well within it's capacity.

No errors reported in console output of either liq or ffmpeg.

Though this is at the top of the script, I guess it is related ?

-edit ... Actually I guess this is not related. [ffmpeg.filter.bitstream:3] No valid mode found for filter pgs_frame_merge!

To Reproduce

v_width = ref(1280)
v_height = ref(720)

settings.frame.video.width.set(v_width())
settings.frame.video.height.set(v_height())

t = ref("title")
a = ref("artist")

coverfile = "cover.jpg"
cover_image = ref(coverfile)

radio = playlist("files")

vid = playlist("video")

vid = source.drop.audio(vid)
# don't need metadata on this video source
# seems to deal with PTS warnings.
vid = source.drop.metadata(vid)

def meta(m)
    title = m['title']
    artist = m['artist']
    if title != "" and title != t() then
        t := title
    end
    if artist != "" and artist != a() then
        a := artist
    end
    vid.skip()
    print("#{t()} - #{a()}")
end

radio.on_metadata(meta)

radio = source.mux.video(video=vid, radio)

##### I've trying migrating the below into the apply filters function but had issues with creating both the rectangle and
##### dynamic text with ffmpeg raw filters.
radio = video.add_rectangle(alpha=0.50, color=0x151522, x=0, y=580, height=100, width=v_width(), radio)
radio = video.add_image(x=10, y=590, width=80, height=80, file=cover_image, radio)
radio = video.add_text(font="font.ttf", color=0xDFDFE1, speed=0, x=100, y=602, size=25,
{"#{t()}"}, radio)
radio = video.add_text(font="font.ttf", color=0xDFDFE1, speed=0, x=100, y=630, size=20,
{"#{a()}"}, radio)

def apply_filters(s) =
  def mkfilter(graph)
    let { audio = audio_track, video = video_track} = source.tracks(s)

    a = ffmpeg.filter.audio.input(graph, audio_track)

    bg = ffmpeg.filter.video.input(graph, video_track)

    let ([a1, a2], _) = ffmpeg.filter.asplit(graph, a)

    v = ffmpeg.filter.showwaves(
      split_channels=false,
      s="800x100",
      mode=3,
      graph, a1)
    
    v = ffmpeg.filter.format(pix_fmts="argb", graph, v)
    v = ffmpeg.filter.colorkey(graph, v)

    v = ffmpeg.filter.overlay(graph, bg, v, x="400", y="580")

    v = ffmpeg.filter.video.output(graph, v)

    # Remove this when https://github.com/savonet/liquidsoap/issues/2384 is closed:
    a = ffmpeg.filter.anull(graph, a2)
    a = ffmpeg.filter.audio.output(graph, a)

    source({
      audio = a,
      video = v
    })
  end

  ffmpeg.filter.create(mkfilter)
end

def myfilter(s)
    s = ffmpeg.raw.encode.audio_video(%ffmpeg(%audio.raw, %video.raw), s)
    s = apply_filters(s)
    s
end

videostream = myfilter(radio)

aac_lofi = 
        %ffmpeg(
    format="mpegts", 
    %video.raw(codec="libx264", pixel_format="yuv420p", b="2500k", preset="superfast", r=25, g=50),
    %audio.raw(
        codec="aac",
        samplerate=44100,
        channels=2,
        b="192k",
        profile="aac_low"
    )
)

streams = [("aac_lofi", aac_lofi.{id3=false})]

def segment_name(~position,~extname,stream_name) =
  timestamp = int_of_float(time())
  duration = 2
  "#{stream_name}_#{duration}_#{timestamp}_#{position}.#{extname}"
end

output.file.hls(playlist="live.m3u8",
                segment_duration=2.0,
                segments=5,
                segments_overhead=5,
                segment_name=segment_name,
                persist_at="/tmp/hls/state.config",
                "/tmp/hls",
                streams,
                fallible=true,
                videostream)

Expected behavior video , text and waveform

Version details

• OS: [e.g. Debian, OSX] - Manjaro
• Version [e.g. 1.3.4] - latest github code (2.3 ?)

Install method Opam

[Warblefly]

Maybe related to #3309? I know our friends with the code will get the root of this.

[Russsgithub]

@Warblefly , thanks.

I think you may be right.

[Russsgithub]

I managed to catch the segmantaion fault with gdb.

Thread 35 "liquidsoap" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffba9936c0 (LWP 81824)]
0x00007ffff4cb7e62 in ?? () from /usr/lib/libc.so.6

Thread 35

Thread 35 (Thread 0x7fffba9936c0 (LWP 81824) "liquidsoap"):
#0  0x00007ffff4cb7e62 in  () at /usr/lib/libc.so.6
#1  0x00007ffff726214a in  () at /usr/lib/libswscale.so.7
#2  0x00007ffff72544b9 in  () at /usr/lib/libswscale.so.7
#3  0x00007ffff7256ec9 in sws_scale () at /usr/lib/libswscale.so.7
#4  0x000055555638a91e in ocaml_swscale_convert (_sws=<optimized out>, _in_vector=<optimized out>) at swscale_stubs.c:343
#5  0x0000555555e8c617 in camlBuiltins_ffmpeg_encoder__fun_3579 () at swscale/swscale.ml:51
#6  0x0000555555f2c3f7 in camlOutput__fun_2355 () at src/core/outputs/output.ml:180
#7  0x0000555555f02335 in camlClock__fun_2360 () at src/core/clock.ml:318
#8  0x000055555626b074 in camlStdlib__List__fold_left_521 () at list.ml:121
#9  0x0000555555eff301 in camlClock__fun_2302 () at src/core/clock.ml:315
#10 0x0000555555f4db3e in camlChild_support__fun_881 () at src/core/tools/child_support.ml:71
#11 0x0000555555f4ea3d in camlProducer_consumer__fun_1812 () at src/core/tools/producer_consumer.ml:125
#12 0x0000555555ef9040 in camlSource__fun_3682 () at src/core/source.ml:649
#13 0x0000555555ef8a67 in camlSource__fun_3801 () at src/core/source.ml:759
#14 0x0000555555f3768c in camlMuxer__fun_2136 () at src/core/operators/muxer.ml:122
#15 0x000055555626afa0 in camlStdlib__List__iter_507 () at list.ml:110
#16 0x0000555555f375b2 in camlMuxer__fun_2139 () at src/core/operators/muxer.ml:139
#17 0x0000555555f37423 in camlMuxer__fun_2142 () at src/core/operators/muxer.ml:161
#18 0x0000555555ef9040 in camlSource__fun_3682 () at src/core/source.ml:649
#19 0x0000555555ef8872 in camlSource__fun_3801 () at src/core/source.ml:776
#20 0x0000555555f2c2e8 in camlOutput__fun_2355 () at src/core/outputs/output.ml:173
#21 0x0000555555f02335 in camlClock__fun_2360 () at src/core/clock.ml:318
#22 0x000055555626b074 in camlStdlib__List__fold_left_521 () at list.ml:121
#23 0x0000555555eff301 in camlClock__fun_2302 () at src/core/clock.ml:315
#24 0x0000555555e80d49 in camlFfmpeg_filter_io__fun_3173 () at src/core/io/ffmpeg_filter_io.ml:229
#25 0x0000555555e80acc in camlFfmpeg_filter_io__fun_3187 () at src/core/io/ffmpeg_filter_io.ml:255
#26 0x0000555555ef9040 in camlSource__fun_3682 () at src/core/source.ml:649
#27 0x0000555555ef8a67 in camlSource__fun_3801 () at src/core/source.ml:759
#28 0x0000555555f3768c in camlMuxer__fun_2136 () at src/core/operators/muxer.ml:122
#29 0x000055555626afa0 in camlStdlib__List__iter_507 () at list.ml:110
#30 0x0000555555f375b2 in camlMuxer__fun_2139 () at src/core/operators/muxer.ml:139
#31 0x0000555555f37423 in camlMuxer__fun_2142 () at src/core/operators/muxer.ml:161
#32 0x0000555555ef9040 in camlSource__fun_3682 () at src/core/source.ml:649
#33 0x0000555555ef8a67 in camlSource__fun_3801 () at src/core/source.ml:759
#34 0x0000555555f2c2e8 in camlOutput__fun_2355 () at src/core/outputs/output.ml:173
#35 0x0000555555f02335 in camlClock__fun_2360 () at src/core/clock.ml:318
#36 0x000055555626b074 in camlStdlib__List__fold_left_521 () at list.ml:121
#37 0x0000555555eff301 in camlClock__fun_2302 () at src/core/clock.ml:315
#38 0x0000555555f0205b in camlClock__loop_1211 () at src/core/clock.ml:277
#39 0x0000555555eff898 in camlClock__fun_2270 () at src/core/clock.ml:280
#40 0x0000555555ee5173 in camlTutils__process_1169 () at src/core/tools/tutils.ml:184
#41 0x000055555622b8f5 in camlThread__fun_850 () at thread.ml:49
#42 0x00005555563eae3d in caml_start_program ()
#43 0x00005555563e108d in caml_callback_exn (closure=closure@entry=140737022629912, arg=<optimized out>, arg@entry=1) at callback.c:111
#44 0x00005555563b48c0 in caml_thread_start (arg=0x555556ce96a0) at st_stubs.c:549
#45 0x00007ffff4c9d44b in  () at /usr/lib/libc.so.6
#46 0x00007ffff4d20e40 in  () at /usr/lib/libc.so.6

I don't know if this is useful , and have the rest of the trace if needed.

......

This is with a buffer added

videostream = myfilter(buffer(buffer=5., radio))

[Moonbase59]

I wonder if there might be some hitherto undiscovered buffer/clock problem. Without being able to exactly pinpoint it, I also experience minor audio glitches in my video stream, when listening for a longer time. This didn’t happen a few versions before.

[Russsgithub]

@Moonbase59 , I guess it could be.

I have a translated script running in ffmpeg / bash which has no problem.

I also notice that mostly only 1 core is being used in liq so maybe the ocaml single threading thing.

I've got no idea to be honest, I guess @toots will have a better idea of possible causes, and will get there when he has time and head space.

Thanks for all the work you put into this @toots and @smimram .

[Russsgithub]

The garbled audio seems to be caused by

ffmpeg.raw.encode.audio_video

and is not helped by adding a buffer.