File based settings

[jooola]

Is your feature request related to a problem? Please describe.

Many settings are editable from the admin UI and are stored in the database.

I suggest to move most of these settings currently in the DB to a dedicated configuration file, or the one we already have. In my case, this allow easier backup/modifications between deployment, for settings that actually does not change that often. This should also simplify other aspects of the configuration handling.

I am mostly thinking about the following settings :

• Server timezone
• Week start day
• CORS
• Public API toggle
• Streams settings
  • Global stream settings
  • Output streams settings
  • Live streaming settings

I we want to go all the way down this road, we might also consider moving some other settings to a file based storage, but we should leave it editable from the admin UI.

[paddatrapper]

This sounds reasonable to me. It does limit who can modify these settings from Admins to the system administrators, but they're settings that shouldn't need changing often anyway

[zklosko]

The stream settings may change when troubleshooting, so they should still be accessible to admins.

[jooola]

The only issue we need to fix to move forward with this, is how to handle handle environment variables when running in a docker environment.

This might become tricky when handling stream settings. We could put a hard requirements on a file based configuration, but I would try to avoid this.

[zklosko]

I don't see the harm extra database reads would be doing. I guess they could be Redis variables if we were going to replace RabbitMQ, but I don't see the need.

[jooola]

This is not a performance issue. Redis and RabbitMQ are irrelevant in this ticket.

[jooola]

From cc_pref

keystr	valstr
timezone	Europe/Paris
allowed_cors_urls	https://station.radio.org
live_stream_master_password	SOME_SECRET
live_stream_master_username	source
live_dj	false
live_dj_source_connection_url	http://stream.radio.org:8802/show
master_dj	false
master_dj_source_connection_url	http://station.radio.org:8801/main

From cc_stream_setting

keyname	value	type
icecast_vorbis_metadata	true	boolean
dj_live_stream_mp	/show	string
dj_live_stream_port	8802	integer
master_live_stream_mp	/main	string
master_live_stream_port	8801	integer
output_sound_device	false	boolean
output_sound_device_type	ALSA	string
s*_admin_pass	SOME_SECRET	string
s*_admin_user	admin	string
s*_bitrate	256	integer
s*_channels	stereo	string
s*_description	Main (ogg 256kbps)	string
s*_enable	true	boolean
s*_genre		string
s*_host	stream.radio.org	string
s*_mobile	0
s*_mount	main.ogg	string
s*_name	Radio	string
s*_output	icecast	string
s*_pass	SOME_SECRET	string
s*_port	8800	integer
s*_type	ogg	string
s*_url	https://www.radio.org	string
s*_user	source	string

To yaml

timezone: Europe/Paris
allowed_hosts:
  - https://station.radio.org
  - https://www.radio.org

live_streams:
  main:
    port: 8001
    mount_point: /main
    username: source
    password: SOME_SECRET
    public_url: https://live-stream.radio.org/main

  show:
    port: 8002
    mount_point: /show
    username: source
    password: SOME_SECRET
    public_url: https://live-stream.radio.org/show

icecast_outputs:
  - enabled: true
    # Meta
    name: Radio
    description: Main (ogg 256kbps)
    url: https://www.radio.org
    genre: various
    # Audio
    format: ogg
    bitrate: 256
    channels: stereo
    # Icecast
    host: localhost
    port: 8000
    mount_point: /main.ogg
    admin_username: admin
    admin_password: SOME_SECRET
    username: source
    password: SOME_SECRET
    public_url: https://stream.radio.org/main.ogg

  - enabled: true
    # Meta
    name: Radio
    description: Main (mp3 256kbps)
    url: https://www.radio.org
    genre: various
    # Audio
    format: mp3
    bitrate: 256
    channels: stereo
    # Icecast
    host: localhost
    port: 8000
    mount_point: /main.mp3
    admin_username: admin
    admin_password: SOME_SECRET
    username: source
    password: SOME_SECRET
    public_url: https://stream.radio.org/main.mp3

[paddatrapper]

Looks good. My only comment would be that some of the settings are optional (e.g. public_url) and I'm not sure how, but we should add validation on the number of icecast outputs supported. Much of the code base (especially liquidsoap) works on the assumption that there are 4 configured with 1 or more active.

[paddatrapper]

Not sure if we want to break it up more:

icecast_outputs:
  - enabled: true
    metadata:
      name: LibreTime
      description: Stream description
      [...]
    audio:
      format: mp3
      bitrate: 256
      channels: 2
    icecast:
      host: localhost
      [...]

[jooola]

Not sure if we want to break it up more:

Agreed, this might help use reuse some component for additional future output, such as hls.

[paddatrapper]

Could even be

outputs:
  - enabled: true
    [...]
    connection:
      type: Icecast
      host: localhost
      [...]

That way we can add support for other streaming providers (e.g. shoutcast) without requiring people to change their configuration files

[jooola]

If I understand the thread correctly, some people seem to rely on the possibility to configure mount points from the UI, but this feels like a hack to accomplish a specific feature.

https://discourse.libretime.org/t/icecast-is-the-admin-user-mandatory-and-why/1169

[paddatrapper]

That doesn't seem like configuring from the UI is required. To me that just seems like their Icecast provider (or their IT department) doesn't want to give them access to the admin Icecast user. Which is reasonable and definitely a use-case we should support, but where that config sits doesn't matter

[jooola]

After talking to some non tech people, I feel like we should provide a way to user to edit some of these configs without forcing them to ssh into the libretime server. Not every radio station has it's own sysadmin and sometimes the admin is not the sysadmin.

I still want to configure the settings using a files, so I think we can allow to read from both sources (config file and database), and mark as read-only some of the settings that are from the config file. This should allow to enforce some settings from the sysadmin side and let the admin configure the rest from the UI.

[paddatrapper]

Another option would be to have the config file be written to by the web server via PHP or the API?

[jooola]

Another option would be to have the config file be written to by the web server via PHP or the API?

If we head in that direction we should convert the config files to json. It will be a bad idea to write yaml config files.

[paddatrapper]

Depends on where the focus should be - if the primary way of editing the file-based settings is by hand, then yaml makes sense. If it is going to be mostly written via the UI, then json

But either way, I think we should define a line between database settings (e.g users) and file-based settings (e.g. the DB connection details). Not all settings should be in the DB and not all settings should be in the file

[jooola]

But either way, I think we should define a line between database settings (e.g users) and file-based settings (e.g. the DB connection details). Not all settings should be in the DB and not all settings should be in the file

Agreed.

[jooola]

Once the file based stream config is merged, I think we could allow power user to define there own outputs using a multiline string that will be inserted in the liquidsoap script on generation. This could allow things like:

stream:
  outputs:
    - kind: manual
      enabled: true
      raw: |
        output.file.hls(
          playlist="live.m3u8",
          segment_duration=2.0,
          segments=5,
          segments_overhead=5,
          segment_name=segment_name,
          persist_at="/path/to/state.config",
          "/path/to/hls/directory",
          streams,
          source
        )

or even include from a liquidsoap file:

stream:
  outputs:
    - kind: manual
      enabled: true
      filepath: /path/to/the/custom/liquidsoap/script.liq
      # Should render an %include "/path/to/the/custom/liquidsoap/script.liq"

Related to #229

[paddatrapper]

That would be a handy way of including that customizability

[jooola]

Should we move the timezone setting to the configuration file ?

[paddatrapper]

Hmm... Personally I think there are too many different timezones in the system - there is a system timezone, a libretime system timezone and a UI timezone configuration. Maybe we drop libretime's timezone configuration have it work in whatever the system's time is and the UI is configurable in the UI? The DB can (and should) still store everything in a timezone-aware manner.

[jooola]

Hmm... Personally I think there are too many different timezones in the system - there is a system timezone, a libretime system timezone and a UI timezone configuration. Maybe we drop libretime's timezone configuration have it work in whatever the system's time is and the UI is configurable in the UI? The DB can (and should) still store everything in a timezone-aware manner.

If I understand it properly, you suggest to get the system timezone automatically ? I don't know if this is flexible enough. If we go that direction, we might want to allow setting libretime timezone and default to automatically fetching from the system, to finally fallback to UTC.

This makes me wonder how we could handle this with django. I feel we should have a setting with a timezone for the libretime backend, as described in https://docs.djangoproject.com/en/4.1/topics/i18n/timezones/. I would also be in favor for turning US_TZ as soon as possible.

I don't mind having 3 different Timezone, the issue might be that django is capable of handling a config based timezone, while other apps default to the system. So we might need to force all the others apps to get the timezone from the config file instead of the system. Otherwise I understand if you want to always fallback to the system timezone and drop the libretime timezone.

[paddatrapper]

Maybe we place it in the config then, as Django needs it before it starts?

[jooola]

Now that #2096 has been merged, this has been resolved.