Getting a lot of random crashes and silent gaps when crossfading. I must be doing something wrong...

[TomKranenburg]

Can anyone take a quick look over this config? The crashes have no error message sadly. So I don't know where to start when it comes to debugging:

log.stdout.set(true)
log.file.set(false)

enable_replaygain_metadata()

settings.server.telnet := true

source = replaygain(playlist("\\\\CITADEL\\Archive\Music\\Playlists\\M3U Playlists\\Ambient - Remote.m3u", reload_mode="watch"))
intermissionssource = replaygain(playlist("\\\\CITADEL\\Archive\Music\\Playlists\\M3U Playlists\\Intermissions - Remote.m3u", reload_mode="watch", mode="randomize"))

radio = mksafe(source)
intermissions = mksafe(intermissionssource)

time_queue = request.queue()

def queue_track(url) = 
  time_queue.push(request.create(url))
end

thread.when(predicate.activates({17h00m}), {list.iter(queue_track, ["\\\\CITADEL\\Archive\\Music\\Music Archive\\Vangelis - Blade Runner Blues (Leviathan Radio Edit).mp3"])})

radio = fallback(track_sensitive=false, [time_queue, radio])

radio = rotate(weights=[1, 3], [intermissions, radio])

fadedradio = crossfade(radio,fade_out=5.0,fade_in=5.0,duration=10.0)

output.icecast(%ffmpeg(format="mp3", %audio(codec="libmp3lame", b="320k")), fallible=true, host = "localhost", port = 8000, password = "hackme", mount = "wave.mp3", fadedradio)

[vitoyucepi]

Hi @TomKranenburg, Could you share the info about your system?

1. What operating system do you use?
2. How did you get liquidsoap? Docker, packages, opam or something else?
3. What version of liquidsoap are you running?
4. How did it look in the logs?

[TomKranenburg]

Hi @TomKranenburg, Could you share the info about your system?

1. What operating system do you use?
2. How did you get liquidsoap? Docker, packages, opam or something else?
3. What version of liquidsoap are you running?
4. How did it look in the logs?

Sorry @vitoyucepi. I should know better. This should have been in my original post.

1. I am running it on Windows 10. I'll migrate the config to a Linux system later.

2. Precompiled binaries @ https://github.com/savonet/liquidsoap/releases/download/v2.2.4/liquidsoap-2.2.4-win64.zip

3. v2.2.4

4. Nothing in the logs my good man. That's why I'm reaching out really. I think if my setup is correct I'm printing the log to stdout anyway? I see nothing when it fully crashes sadly.

I'll add that the music files are being accessed via liquidsoap over an SMB share. I do wonder if that's part of the problem. In the logs I'm getting a lot of things like this:

2024/03/04 17:39:43 [clock.main:2] We must catchup 5.16 seconds!

This is generally when the skips occur I think.

[TomKranenburg]

I will add as well. I'm getting a lot of:

[mp3 @ 000001b3d2ed7840] Estimating duration from bitrate, this may be inaccurate

This is going to be a problem I think as a lot of my MP3s are VBR.

Thread about it here:

https://stackoverflow.com/questions/10437750/how-to-get-the-real-actual-duration-of-an-mp3-file-vbr-or-cbr-server-side#10571671

Looks like it's down to ffmpeg.

[vitoyucepi]

Could you verify that the problem is present in rolling-release-v2.2.x? You can download the binaries from the release page.

v2.2.4

There's a problem here, there are two versions 2.2.4 and 2.2.4-1. I think if you run liquidsoap.exe --build-config, it will print all the necessary information.

Build config

 * Liquidsoap version  : 2.2.4-1

 * Compilation options
   - Release build       : true
   - Git SHA             : (none)
   - OCaml version       : 4.14.1
   - OS type             : Win32
   - Libs versions       : angstrom=0.15.0 ao=0.2.4 bigarray=[distributed with OCaml] bigstringaf=0.9.0 bytes=[distributed with OCaml] camlp-streams camomile.lib=2.0 cry=1.0.2 ctypes=0.17.1 ctypes.foreign=0.17.1 ctypes.foreign.base=0.17.1 ctypes.foreign.threaded=0.17.1 ctypes.stubs=0.17.1 curl=0.9.2 dtools=0.4.5 dune-build-info=3.2.0 dune-private-libs.dune-section=3.2.0 dune-site=3.2.0 duppy=0.9.3 ffmpeg-av=1.1.10 ffmpeg-avcodec=1.1.10 ffmpeg-avdevice=1.1.10 ffmpeg-avfilter=1.1.10 ffmpeg-avutil=1.1.10 ffmpeg-swresample=1.1.10 ffmpeg-swscale=1.1.10 fileutils=0.6.4 gen=1.1 integers lastfm=0.3.3 liquidsoap-lang=2.2.4-1 liquidsoap-lang.console=2.2.4-1 liquidsoap_ao liquidsoap_builtins liquidsoap_core liquidsoap_ffmpeg liquidsoap_lastfm liquidsoap_mem_usage liquidsoap_memtrace liquidsoap_optionals liquidsoap_portaudio liquidsoap_runtime liquidsoap_samplerate liquidsoap_srt liquidsoap_ssl liquidsoap_stereotool liquidsoap_taglib liquidsoap_xmlplaylist magic-mime=1.3.1 mem_usage=0.0.3 memtrace=v0.2.2 menhirLib=20210929 metadata=0.2.0 mm=0.8.4 mm.audio=0.8.4 mm.base=0.8.4 mm.image=0.8.4 mm.midi=0.8.4 mm.video=0.8.4 pcre=7.5.0 portaudio=0.2.3 posix-base=2.0.0 posix-socket=2.0.0 posix-socket.constants=2.0.0 posix-socket.stubs=2.0.0 posix-socket.types=2.0.0 samplerate=0.1.6 sedlex=3.2 seq=[distributed with OCaml 4.07 or above] srt=0.3.0 srt.constants=0.3.0 srt.stubs=0.3.0 srt.stubs.locked=0.3.0 srt.types=0.3.0 ssl=0.7.0 stdlib-shims=0.3.0 stereotool str=[distributed with OCaml] stringext=1.6.0 taglib=0.3.10 threads=[distributed with OCaml] threads.posix=[distributed with OCaml] unix=[distributed with OCaml] uri=4.2.0 winsvc=1.0.1 xmlm=v1.3.0 xmlplaylist=0.1.5
   - architecture        : amd64
   - host                : x86_64-w64-mingw32.static
   - target              : x86_64-w64-mingw32.static
   - system              : mingw64
   - ocamlopt_cflags     : -O2 -fno-strict-aliasing -fwrapv -mms-bitfields
   - native_c_compiler   : x86_64-w64-mingw32.static-gcc -O2 -fno-strict-aliasing -fwrapv -mms-bitfields -D_FILE_OFFSET_BITS=64
   - native_c_libraries  : -lws2_32 -lversion

 * Configured paths
   - mode              : standalone
   - standard library  : ./libs
   - scripted binaries : ./bin
   - rundir            : ./run
   - logdir            : ./log
   - camomile files    : ./camomile

 * Supported input formats
   - MP3               : no (requires mad)
   - AAC               : no (requires faad)
   - Ffmpeg            : yes
   - Flac (native)     : no (requires flac)
   - Flac (ogg)        : no (requires ogg)
   - Opus              : no (requires opus)
   - Speex             : no (requires speex)
   - Theora            : no (requires theora)
   - Vorbis            : no (requires vorbis)

 * Supported output formats
   - FDK-AAC           : no (requires fdkaac)
   - Ffmpeg            : yes
   - MP3               : no (requires lame)
   - MP3 (fixed-point) : no (requires shine)
   - Flac (native)     : no (requires flac)
   - Flac (ogg)        : no (requires ogg)
   - Opus              : no (requires opus)
   - Speex             : no (requires speex)
   - Theora            : no (requires theora)
   - Vorbis            : no (requires vorbis)

 * Tags
   - Taglib (ID3 tags) : yes
   - Vorbis            : no (requires vorbis)

 * Input / output
   - ALSA              : no (requires alsa)
   - AO                : yes
   - FFmpeg            : yes
   - GStreamer         : no (requires gstreamer)
   - JACK              : no (requires bjack)
   - OSS               : no (requires oss)
   - Portaudio         : yes
   - Pulseaudio        : no (requires pulseaudio)
   - SRT               : yes

 * Audio manipulation
   - FFmpeg            : yes
   - LADSPA            : no (requires ladspa)
   - Lilv              : no (requires lilv)
   - Samplerate        : yes
   - SoundTouch        : no (requires soundtouch)
   - StereoTool        : yes

 * Video manipulation
   - camlimages        : no (requires camlimages)
   - FFmpeg            : yes
   - frei0r            : no (requires frei0r)
   - ImageLib          : no (requires imagelib)
   - SDL               : no (requires tsdl-image & tsdl-ttf)

 * MIDI manipulation
   - DSSI              : no (requires dssi)

 * Visualization
   - GD                : no (requires gd)
   - Graphics          : no (requires graphics)
   - SDL               : no (requires tsdl-image & tsdl-ttf)

 * Additional libraries
   - FFmpeg filters    : yes
   - FFmpeg devices    : yes
   - inotify           : no (requires inotify)
   - irc               : no (requires irc-client-unix)
   - jemalloc          : no (requires jemalloc)
   - lastfm            : yes
   - lo                : no (requires lo)
   - memtrace          : yes
   - mem_usage         : yes
   - osc               : no (requires osc-unix)
   - ssl               : yes
   - tls               : no (requires tls-liquidsoap)
   - posix-time2       : no (requires posix)
   - windows service   : yes
   - YAML support      : no (requires yaml)
   - XML playlists     : yes

 * Monitoring
   - Prometheus        : no (requires prometheus)

Also, right after the start liquidsoap will print long line with the libs versions, you could post it instead.

Libs

angstrom=0.15.0 ao=0.2.4 bigarray=[distributed with OCaml] bigstringaf=0.9.0 bytes=[distributed with OCaml] camlp-streams camomile.lib=2.0 cry=1.0.2 ctypes=0.17.1 ctypes.foreign=0.17.1 ctypes.foreign.base=0.17.1 ctypes.foreign.threaded=0.17.1 ctypes.stubs=0.17.1 curl=0.9.2 dtools=0.4.5 dune-build-info=3.2.0 dune-private-libs.dune-section=3.2.0 dune-site=3.2.0 duppy=0.9.3 ffmpeg-av=1.1.10 ffmpeg-avcodec=1.1.10 ffmpeg-avdevice=1.1.10 ffmpeg-avfilter=1.1.10 ffmpeg-avutil=1.1.10 ffmpeg-swresample=1.1.10 ffmpeg-swscale=1.1.10 fileutils=0.6.4 gen=1.1 integers lastfm=0.3.3 liquidsoap-lang=2.2.4-1 liquidsoap-lang.console=2.2.4-1 liquidsoap_ao liquidsoap_builtins liquidsoap_core liquidsoap_ffmpeg liquidsoap_lastfm liquidsoap_mem_usage liquidsoap_memtrace liquidsoap_optionals liquidsoap_portaudio liquidsoap_runtime liquidsoap_samplerate liquidsoap_srt liquidsoap_ssl liquidsoap_stereotool liquidsoap_taglib liquidsoap_xmlplaylist magic-mime=1.3.1 mem_usage=0.0.3 memtrace=v0.2.2 menhirLib=20210929 metadata=0.2.0 mm=0.8.4 mm.audio=0.8.4 mm.base=0.8.4 mm.image=0.8.4 mm.midi=0.8.4 mm.video=0.8.4 pcre=7.5.0 portaudio=0.2.3 posix-base=2.0.0 posix-socket=2.0.0 posix-socket.constants=2.0.0 posix-socket.stubs=2.0.0 posix-socket.types=2.0.0 samplerate=0.1.6 sedlex=3.2 seq=[distributed with OCaml 4.07 or above] srt=0.3.0 srt.constants=0.3.0 srt.stubs=0.3.0 srt.stubs.locked=0.3.0 srt.types=0.3.0 ssl=0.7.0 stdlib-shims=0.3.0 stereotool str=[distributed with OCaml] stringext=1.6.0 taglib=0.3.10 threads=[distributed with OCaml] threads.posix=[distributed with OCaml] unix=[distributed with OCaml] uri=4.2.0 winsvc=1.0.1 xmlm=v1.3.0 xmlplaylist=0.1.5

[mp3 @ 000001b3d2ed7840] Estimating duration from bitrate, this may be inaccurate

You can't calculate the exact duration just by closely looking at the file, you have to decode the whole file. So I think this is not a problem.

[TomKranenburg]

Ok here is the liquidsoap.exe --build-config output:

Output

``` * Liquidsoap version : 2.2.4-1

• Compilation options

  • Release build : true
  • Git SHA : (none)
  • OCaml version : 4.14.1
  • OS type : Win32
  • Libs versions : angstrom=0.15.0 ao=0.2.4 bigarray=[distributed with OCaml] bigstringaf=0.9.0 bytes=[distributed with OCaml] camlp-streams camomile.lib=2.0 cry=1.0.2 ctypes=0.17.1 ctypes.foreign=0.17.1 ctypes.foreign.base=0.17.1 ctypes.foreign.threaded=0.17.1 ctypes.stubs=0.17.1 curl=0.9.2 dtools=0.4.5 dune-build-info=3.2.0 dune-private-libs.dune-section=3.2.0 dune-site=3.2.0 duppy=0.9.3 ffmpeg-av=1.1.10 ffmpeg-avcodec=1.1.10 ffmpeg-avdevice=1.1.10 ffmpeg-avfilter=1.1.10 ffmpeg-avutil=1.1.10 ffmpeg-swresample=1.1.10 ffmpeg-swscale=1.1.10 fileutils=0.6.4 gen=1.1 integers lastfm=0.3.3 liquidsoap-lang=2.2.4-1 liquidsoap-lang.console=2.2.4-1 liquidsoap_ao liquidsoap_builtins liquidsoap_core liquidsoap_ffmpeg liquidsoap_lastfm liquidsoap_mem_usage liquidsoap_memtrace liquidsoap_optionals liquidsoap_portaudio liquidsoap_runtime liquidsoap_samplerate liquidsoap_srt liquidsoap_ssl liquidsoap_stereotool liquidsoap_taglib liquidsoap_xmlplaylist magic-mime=1.3.1 mem_usage=0.0.3 memtrace=v0.2.2 menhirLib=20210929 metadata=0.2.0 mm=0.8.4 mm.audio=0.8.4 mm.base=0.8.4 mm.image=0.8.4 mm.midi=0.8.4 mm.video=0.8.4 pcre=7.5.0 portaudio=0.2.3 posix-base=2.0.0 posix-socket=2.0.0 posix-socket.constants=2.0.0 posix-socket.stubs=2.0.0 posix-socket.types=2.0.0 samplerate=0.1.6 sedlex=3.2 seq=[distributed with OCaml 4.07 or above] srt=0.3.0 srt.constants=0.3.0 srt.stubs=0.3.0 srt.stubs.locked=0.3.0 srt.types=0.3.0 ssl=0.7.0 stdlib-shims=0.3.0 stereotool str=[distributed with OCaml] stringext=1.6.0 taglib=0.3.10 threads=[distributed with OCaml] threads.posix=[distributed with OCaml] unix=[distributed with OCaml] uri=4.2.0 winsvc=1.0.1 xmlm=v1.3.0 xmlplaylist=0.1.5
  • architecture : amd64
  • host : x86_64-w64-mingw32.static
  • target : x86_64-w64-mingw32.static
  • system : mingw64
  • ocamlopt_cflags : -O2 -fno-strict-aliasing -fwrapv -mms-bitfields
  • native_c_compiler : x86_64-w64-mingw32.static-gcc -O2 -fno-strict-aliasing -fwrapv -mms-bitfields -D_FILE_OFFSET_BITS=64
  • native_c_libraries : -lws2_32 -lversion

• Configured paths

  • mode : standalone
  • standard library : ./libs
  • scripted binaries : ./bin
  • rundir : ./run
  • logdir : ./log
  • camomile files : ./camomile

• Supported input formats

  • MP3 : no (requires mad)
  • AAC : no (requires faad)
  • Ffmpeg : yes
  • Flac (native) : no (requires flac)
  • Flac (ogg) : no (requires ogg)
  • Opus : no (requires opus)
  • Speex : no (requires speex)
  • Theora : no (requires theora)
  • Vorbis : no (requires vorbis)

• Supported output formats

  • FDK-AAC : no (requires fdkaac)
  • Ffmpeg : yes
  • MP3 : no (requires lame)
  • MP3 (fixed-point) : no (requires shine)
  • Flac (native) : no (requires flac)
  • Flac (ogg) : no (requires ogg)
  • Opus : no (requires opus)
  • Speex : no (requires speex)
  • Theora : no (requires theora)
  • Vorbis : no (requires vorbis)

• Tags

  • Taglib (ID3 tags) : yes
  • Vorbis : no (requires vorbis)

• Input / output

  • ALSA : no (requires alsa)
  • AO : yes
  • FFmpeg : yes
  • GStreamer : no (requires gstreamer)
  • JACK : no (requires bjack)
  • OSS : no (requires oss)
  • Portaudio : yes
  • Pulseaudio : no (requires pulseaudio)
  • SRT : yes

• Audio manipulation

  • FFmpeg : yes
  • LADSPA : no (requires ladspa)
  • Lilv : no (requires lilv)
  • Samplerate : yes
  • SoundTouch : no (requires soundtouch)
  • StereoTool : yes

• Video manipulation

  • camlimages : no (requires camlimages)
  • FFmpeg : yes
  • frei0r : no (requires frei0r)
  • ImageLib : no (requires imagelib)
  • SDL : no (requires tsdl-image & tsdl-ttf)

• MIDI manipulation

  • DSSI : no (requires dssi)

• Visualization

  • GD : no (requires gd)
  • Graphics : no (requires graphics)
  • SDL : no (requires tsdl-image & tsdl-ttf)

• Additional libraries

  • FFmpeg filters : yes
  • FFmpeg devices : yes
  • inotify : no (requires inotify)
  • irc : no (requires irc-client-unix)
  • jemalloc : no (requires jemalloc)
  • lastfm : yes
  • lo : no (requires lo)
  • memtrace : yes
  • mem_usage : yes
  • osc : no (requires osc-unix)
  • ssl : yes
  • tls : no (requires tls-liquidsoap)
  • posix-time2 : no (requires posix)
  • windows service : yes
  • YAML support : no (requires yaml)
  • XML playlists : yes

• Monitoring

  • Prometheus : no (requires prometheus)

</details>

[MaPePeR]

Do you have files with umlauts in your playlist and are your playlists UTF-8 encoded? If not - try to encode them as UTF-8.

[TomKranenburg]

Do you have files with umlauts in your playlist and are your playlists UTF-8 encoded? If not - try to encode them as UTF-8.

They are UTF-8 for sure. I think there this no umlauts. I try to stick to ASCII for file names. There will be non ASCII in the MP3 tags I'm sure of it.

[toots]

Thanks for the report. More details about the crash would help. Are you getting any log at all?

We have tracked down segfaults in the ffmpeg code when opening files that have non-utf8 characters on windows. Can that be linked? Also, have you tried without samba in the mix?

[TomKranenburg]

Thanks for the report. More details about the crash would help. Are you getting any log at all?

We have tracked down segfaults in the ffmpeg code when opening files that have non-utf8 characters on windows. Can that be linked? Also, have you tried without samba in the mix?

Perfect timing.

Actually yes I did try it without Samba in the mix. I created a system that copies every single track to a local SSD to see if that could help. Anticipating access times on a network file system must be a horror show.

However it changed absolutely nothing in the slightest. I should add that the time it has to catch up is almost always around 5.15 seconds. It's pretty consistently close to this if that helps. Here's some excerpts:

2024/03/09 07:20:42 [clock.main:2] We must catchup 5.15 seconds!
2024/03/09 07:24:20 [clock.main:2] We must catchup 5.18 seconds!
2024/03/09 07:26:08 [clock.main:2] We must catchup 5.18 seconds!
2024/03/09 07:32:32 [clock.main:2] We must catchup 5.13 seconds!
2024/03/09 07:33:58 [clock.main:2] We must catchup 5.16 seconds!

We have tracked down segfaults in the ffmpeg code when opening files that have non-utf8 characters on windows. Can that be linked?

It could be. Only the file names are clean, the actual metadata will have all kinds of non standard characters. There is nothing in the log. From my perspective it's a silent fail. But next time it tanks I'll make a note of the file names queued and see if they have any odd metadata. I'll then update here.

Thanks for this. As a software package I think liquidsoap is great. I'm not giving it up.

[TomKranenburg]

I just want to add,

I actually upgraded the CPU on this system without reinstalling the OS. I felt this was a risky manoeuvre personally. Many will fight me on that however and say it's totally fine. So far there have been no issues for months but this could be the first one couldn't it.

Just wanted to add that as the time lag there is unusually consistent.