Regex variable not found

[cinderblockgames]

Is there a way to take only the first X lines of the video description?

[jmbannon]

You can use regex to capture the first X lines, something like:

  my_example_preset:
    regex:

      from:
        description:
          match:
           - "(.*)\\n"
          capture_group_names:
            - "shortened_description"
          capture_group_defaults:
            - "{description}"

[cinderblockgames]

This was working, but now I'm getting this:

[ytdl-sub] Validation error in gmm.overrides.description: Format variable 'shortened_description' does not exist.

I have this in the config:

  only_first_line_description:

    regex:
      from:
        description:
          match:
           - "(.*)\\n"
          capture_group_names:
          - "shortened_description"
          capture_group_defaults:
          - "{description}"

And this in the overrides:

  overrides:
    description: "{shortened_description}"

Any thoughts?

[jmbannon]

@cinderblockgames can you post your full config and subscriptions

[cinderblockgames]

Sure.

config.yaml

# Set the working directory which will be used to stage downloads
# before placing them in your desired output directory.
configuration:
  working_directory: '/data/media/youtube/.downloads'

# Presets are where you create 'sub-configs' that can can be
# merged together to dictate what is downloaded, how to format it,
# and what metadata to generate.
presets:

  # Let us create a preset called `only_recent_videos` that will
  # only download recent videos in the last 2 months.
  only_recent_videos:

    # Use the `date_range` plugin to specify ytdl-sub to only
    # download videos after today MINUS {download_range}, which
    # is an override variable that we can alter per channel.
    date_range:
      after: "today-{download_range}"

    # Any yt-dlp argument can be passed via ytdl-sub. Let us set
    # yt-dlp's `break_on_reject` to True to stop downloading after
    # any video is rejected. Videos will be rejected if they are
    # uploaded after our {download_range}.
    ytdl_options:
      break_on_reject: True

    # Deletes any videos uploaded after {download_range}.
    output_options:
      keep_files_after: "today-{download_range}"

    # Set the override variable {download_range} to 2months.
    # This will serve as our default value. We can override
    # this per channel or in a child preset.
    overrides:
      download_range: "2months"

  ####################################################################

  # Now let us create a preset that downloads videos and formats
  # as TV shows.
  tv_show:

    # Presets can inherit all attributes from other presets. Our
    # `tv_show` preset will inherit these presets built into ytdl-sub.
    preset:
      # Let us specify all the TV show by date presets to support all
      # players. You only need to specify one, but this ensures
      # compatibility with all players.
      #- "kodi_tv_show_by_date"
      - "jellyfin_tv_show_by_date"
      #- "plex_tv_show_by_date"
      # Now we choose a preset that defines how our seasons and
      # episode numbers look.
      - "season_by_year__episode_by_month_day"

    # Set override variables that will be applicable to all downloads
    # in main presets.
    overrides:
      tv_show_directory: "/data/media/youtube"  # Replace with desired directory

  ####################################################################
  # https://github.com/jmbannon/ytdl-sub/issues/582#issuecomment-1499301493

  only_first_line_description:

    regex:
      from:
        description:
          match:
           - "(.*)\\n"
          capture_group_names:
          - "shortened_description"
          capture_group_defaults:
          - "{description}"

  ####################################################################
  # https://github.com/jmbannon/ytdl-sub/blob/master/examples/music_audio_config.yaml

  audio_base:
    # Store all music under our music_directory (to be set as an override variable).
    # Store each resulting file with its full track path, and treat thumbnails as the album covers.
    #
    # Maintain a download archive. This will produce a hidden file in your music directory
    # containing an archive of all audio previously downloaded. This makes it so we do not
    # re-download files we already have.
    output_options:
      output_directory: "{music_directory}"
      file_name: "{track_full_path}"
      thumbnail_name: "{album_cover_path}"
      maintain_download_archive: True

    # Set break_on_existing to True. This will stop fetching any more metadata once we
    # hit a video/audio that we already have downloaded.
    ytdl_options:
      break_on_existing: True

    # Extract any audio from files using this codec and quality.
    audio_extract:
      codec: "mp3"
      quality: 320

    # Set these music tags on every resulting audio file.
    # It is recommended to keep most of this as-is, and use override
    # variables to set them to be what you want.
    music_tags:
      tags:
        artist: "{track_artist}"
        artists: "{track_artist}"
        albumartist: "{track_artist}"
        albumartists: "{track_artist}"
        title: "{track_title}"
        album: "{track_album}"
        track: "{track_number}"
        tracktotal: "{track_total}"
        year: "{track_year}"
        genre: "{track_genre}"

    # For every configurable field, make it an override variable,
    # so we can carefully tune different use-cases by only modifying override variables.
    overrides:
      # Track Overrides. By default, set overrides to make each song its own album
      track_title: "{title}"
      track_album: "{title}"
      track_artist: "{channel}"
      track_number: "1"
      track_number_padded: "01"
      track_total: "1"
      track_year: "{upload_year}"
      track_genre: "Unset"
      # Filename Overrides
      track_file_name: "{track_number_padded} - {track_title_sanitized}.{ext}"
      album_file_name: "folder.{thumbnail_ext}"
      # Directory Name Overrides
      music_directory: "/data/media/youtube-audio"
      artist_dir: "{track_artist_sanitized}"
      album_dir: "[{track_year}] {track_album_sanitized}"
      # Full Filepath Overrides
      track_full_path: "{artist_dir}/{album_dir}/{track_file_name}"
      album_cover_path: "{artist_dir}/{album_dir}/{album_file_name}"

  # Make the 'single` preset accept a single URL using the override variable 'url'
  # Each audio file will reside in its own album.
  audio_single:
    # Inherit from `base`
    preset: "audio_base"

    download:
      download_strategy: "url"
      url: "{url}"

  # Make the 'albums_from_playlists' preset format audio files to reside under albums, where
  # each album is a playlist. If a file downloaded using this preset is not part of a playlist,
  # it will default to how it'd look as a `single`.
  albums_from_playlists:
    # Inherit from single
    preset: "audio_single"

    # Override various track properties using playlist variables.
    overrides:
      track_album: "{playlist_title}"
      track_number: "{playlist_index}"
      track_number_padded: "{playlist_index_padded}"
      track_total: "{playlist_count}"
      track_year: "{playlist_max_upload_year}"

subscriptions.yaml

# https://github.com/jmbannon/ytdl-sub#subscriptions

gmm:
  preset:
    - "tv_show"
    - "only_recent_videos"
    - "only_first_line_description"
  overrides:
    tv_show_name: "Good Mythical Morning"
    url: "https://www.youtube.com/@GoodMythicalMorning"
    description: "{shortened_description}"

hot_ones:
  preset:
    - "tv_show"
    - "only_first_line_description"
  overrides:
    tv_show_name: "Hot Ones"
    url: "https://m.youtube.com/playlist?list=PLAzrgbu8gEMIIK3r4Se1dOZWSZzUSadfZ"
    description: "{shortened_description}"

#sorted:
#  preset:
#    - "tv_show"
#    - "only_recent_videos"
#  overrides:
#    tv_show_name: "Sorted Food"
#    url: "https://www.youtube.com/@SortedFood"

# https://github.com/jmbannon/ytdl-sub/blob/master/examples/music_audio_subscriptions.yaml

#drawfee_drawtectives:
#  preset: "albums_from_playlists"
#  overrides:
#    url: "https://www.youtube.com/playlist?list=PLDUoOYLKP1C7ckRGiX3SsLNeAQbwC-R2X"

[jmbannon]

Looks like a valid bug. Can you try using this instead of overriding description:

episode_plot: "{webpage_url}\n\n{shortened_description}"

[cinderblockgames]

It looks like that works.

[Svagtlys]

I wanted to add that I believe I'm seeing a similar issue when trying to set a regex-matched capture group to the "title" variable in overrides.

config.yaml

    ###########################################################################
    # Top-level configurations to apply umask and persist error logs
    configuration:
      umask: "002"
      persist_logs:
        logs_directory: './logs'
        keep_successful_logs: False
      working_directory: '/config/.ytdl-sub-downloads'
      dl_aliases:
        tv: '--preset "TV Show Only Recent"'
        anime: '--preset "Anime Crunchyroll"'
        url: "--overrides.url"
        name: "--overrides.tv_show_name"
        images: "--ytdl_options.skip_download True"

    presets:

      ###############################################################################
      # Set tv_show_directory here instead of in the subscriptions file
      tv_show_paths:
        overrides:
          tv_show_directory: "/tv_shows"

      ###############################################################################
      # Filter out any YouTube shorts
      no_shorts:
        match_filters:
          filters:
            - "original_url!*=/shorts/"

      ###############################################################################
      # Remove all the following sponsorblock sections
      sponsorblock:
        chapters:
          sponsorblock_categories:
            - "outro"
            - "selfpromo"
            - "preview"
            - "interaction"
            - "sponsor"
            - "music_offtopic"
            - "intro"
          remove_sponsorblock_categories: "all"
          force_key_frames: False

      ###############################################################################
      # Wait 2 days before downloading in hopes to get more accurate sponsorblock
      sponsorblock_wait:
        # Import the sponsorblock preset defined above
        preset:
          - "sponsorblock"

        date_range:
          before: "today-2days"


      ###############################################################################
      # base preset to use on all TV Show-based subscriptions
      TV Show Basic:
        preset:
          - "Kodi TV Show by Date"  # Set intended player
          - "best_video_quality"    # prebuilt preset to get best quality
          - "tv_show_paths"

        # Embed chapters into video files
        chapters:
          embed_chapters: True

        # Embed English subtitles into video files (supports more)
        subtitles:
          embed_subtitles: True
          languages:
            - "en"
          allow_auto_generated_subtitles: True

        # ytdl_options lets you pass any arg into yt-dlp's Python API
        ytdl_options:
          # Set the cookie file
          # cookiefile: "/config/youtube_cookies.txt"
          format:  "bv*[height<=1080]+ba/b[height<=1080] / wv*+ba/w"
          break_on_existing: True

          # For YouTube, get English metadata if multiple languages are present
          extractor_args:
            youtube:
              lang:
                - "en"
        nfo_tags:
          tags:
            title: "{file_title}"
        
        overrides:
          subscription_indent_1: "Uncategorized"
          # subscription_indent_2: "default-content-rating"

          tv_show_name: "{subscription_name}"
          url: "{subscription_value}"
          genre: "{subscription_indent_1}"
          # content_rating: "{subscription_indent_2}"    

      ###############################################################################
      # Custom preset to archive an entire channel
      TV Show Full Archive:
        preset:
          - "TV Show Basic"
          - "sponsorblock_wait"  # wait for sponsorblock when full-archiving

      ###############################################################################
      # Custom preset to only fetch and keep recent videos.
      # Format the videos in reverse order, so the first video is the most recent
      # Also include the prebuilt "Only Recent" preset
      TV Show Only Recent:
        preset:
          - "TV Show Basic"
          - "sponsorblock"
          - "no_shorts"
          - "season_by_year__episode_by_month_day_reversed"
          - "Only Recent"

        overrides:
          date_range: "2months"

      ###############################################################################
      # Custom preset to fetch Crunchyroll Anime

      Anime Crunchyroll:
        preset:
          - "TV Show Basic"

        # Embed English subtitles into video files (supports more)
        subtitles:
          embed_subtitles: True
          languages:
            - "en-US"

        # ytdl_options lets you pass any arg into yt-dlp's Python API
        ytdl_options:
          flat-playlist: True
          rejecttitle: ".*(dub|Dub).*"
          cookiefile: "/tv_shows/cookies.txt"

        regex:
          skip_if_match_fails: True
          from:
            title:
              match: 
                # - '^(?:.*?)(?= Episode )(?: Episode )(\\d{1,3})(?:\\W{3})(.*?)(?=$).*'
                - '^.* Episode (\d{1,3})\W{3}(.*)'
              capture_group_names:
                - "captured_episode"
                - "captured_title"
              capture_group_defaults:
                - "{episode_number}"
                - "{title}"

        overrides:
          title: "{captured_title}"
          episode_number: "{captured_episode}"

      ###############################################################################
      # Custom preset to only fetch and keep recent videos.
      # Format the videos in reverse order, so the first video is the most recent
      # Also include the prebuilt "Only Recent" preset
      TV Show Very Recent:
        preset:
          - "TV Show Only Recent"

        overrides:
          date_range: "2weeks"

subscriptions.yaml

Anime Crunchyroll:
  = Adventure:
    "Spy X Family": 
      overrides:
        subscription_value: "https://www.crunchyroll.com/series/G4PH0WXVJ/spy-x-family"

Getting the error:

ytdl_sub.utils.exceptions.StringFormattingVariableNotFoundException: Validation error in Spy X Family.overrides.title: Format variable 'captured_title' does not exist.

Changing title to file_title resolves this error.

[jmbannon]

@Svagtlys Set it to episode_title: "{captured_title}", that is what the preset uses and will show properly in Kodi.

Soon I will add validation to not be able to override 'native' fields like title