Media Extractor integration suddenly stopped working

[anachronox99]

The problem

I have been using Media Extractor integration to play a youtube video to my Google Nest display. Been working fine, but suddenly has stopped working. I haven't changed anything as far as I can tell in my configuration, however I have been keeping my HA installation up to date, so maybe a recent update may be related to this.

A simple test of this by calling a service fails when I call this from the Developer tools UI.

service: media_extractor.play_media data: media_content_id: https://www.youtube.com/watch?v=fqI6aZLR6ng&ab_channel=SilvermanSoundStudios media_content_type: VIDEO target: entity_id: media_player.kitchen_display

I've tried other youtube files but no luck. I've also tried casting videos to my Display using my phone and computer and that works fine. I've also restarted both HA and my Google nest display.

Any ideas would be welcome. Perhaps

What version of Home Assistant Core has the issue?

Home Assistant 2023.2.5 Supervisor 2023.01.1 Operating System 9.5 Frontend 20230202.0 - latest

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Media Extractor

Link to integration documentation on our website

https://www.home-assistant.io/integrations/media_extractor/

Diagnostics information

No response

Example YAML snippet

service: media_extractor.play_media
data:
  media_content_id: https://www.youtube.com/watch?v=fqI6aZLR6ng&ab_channel=SilvermanSoundStudios
  media_content_type: VIDEO
target:
  entity_id: media_player.kitchen_display

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant[bot]]

media_extractor documentation media_extractor source

[lwsrbrts]

You'll probably see this in your logs too.

ERROR: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Looks like a YouTube metadata change causing the issue. Apparently fixed in the library now but assume it needs porting to HA's implementation.

https://github.com/ytdl-org/youtube-dl/issues/31530

[anachronox99]

Thanks! Glad it's fixed. What happens next - is this integration still being supported / will it get ported to HA's implementation?

[minhphung1002]

Thanks! Glad it's fixed. What happens next - is this integration still being supported / will it get ported to HA's implementation?

Please let me know how to fix the error. Currently, I am using the latest version of HA. Thank you so much! Home Assistant 2023.2.5 Supervisor 2023.01.1 Frontend 20230202.0 - latest

[lwsrbrts]

Unfortunately, I only know as much as you do - I'm not a developer but just wondered why my use of Media Extractor stopped working.

I would hope it does get ported over but I guess we'll find out in a few days! :)

[casaoui86]

Same here I did not changed anything,

I also updated to the last version oh HA but same error

Here the version of my system : Home Assistant 2023.2.5 Supervisor 2023.01.1 Operating System 9.5 Frontend 20230202.0 - latest

Logger: homeassistant.components.media_extractor
Source: components/media_extractor/__init__.py:48
Integration: Media Extractor ([documentation](https://www.home-assistant.io/integrations/media_extractor), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+media_extractor%22))
First occurred: 2:54:15 PM (2 occurrences)
Last logged: 3:07:27 PM

Could not retrieve data for the URL: https://m.youtube.com/watch?v=X-9Nmb9hPMs
Could not retrieve data for the URL: https://m.youtube.com/watch?v=xRoaPdBcxAU

[Scags104]

Is there any inkling of an ETA for a fix here? I know its reliant on outside projects.

Kids go-to sleep music is lacking its automation.

[nitobuendia]

Since there's no ETA, sharing a sort of "workaround" (specially for those using media_extractor for YouTube audio) in case it's helpful:

1. Download the YouTube videos
2. If needed, convert to the right format (e.g. mp3)
3. Host video/audio locally on www/ folder
4. Use media_player.play_media instead of media_extractor.media_player to play the local files.

---

This is how it worked for me:

1. Download the YouTube videos (as audio)
2. Convert audio from webm to mp3

downloader.sh

array=(
  dvgZkm1xWPE
  iPUmE-tne5U
  IcrbM1l_BoI
  # Other YouTube video ids
)

for i in "${array[@]}"
do
  if [ ! -f "$(pwd)/$i.mp3" ]; then
    docker run --rm -i -v "$(pwd)":/workdir:rw mikenye/youtube-dl $i --format "bestaudio" --audio-format mp3 --output "/workdir/%(id)s.%(ext)s"
    ffmpeg -i "$i.webm" -vn -ab 128k -ar 44100 -y "$i.mp3"; # TODO: Replace with docker container.
  else
    echo "File $i.mp3 exists, skipping."
  fi
done

ls

This requires docker and ffmpeg installed on the machine.

3. Uploaded the mp3 files on www/alarm/ folder. This will serve on https://myHassio/alarm/videoId.mp3

4. Then, I changed my Node-RED flow to first try using media_extractor; if that fails, I use the local folder with media_player.

[Scags104]

Since there's no ETA, sharing a sort of "workaround" (specially for those using media_extractor for YouTube audio) in case it's helpful:

1. Download the YouTube videos
2. If needed, convert to the right format (e.g. mp3)
3. Host video/audio locally on www/ folder
4. Use media_player.play_media instead of media_extractor.media_player to play the local files.

This is how it worked for me:

1. Download the YouTube videos (as audio)
2. Convert audio from webm to mp3

downloader.sh

array=(
  dvgZkm1xWPE
  iPUmE-tne5U
  IcrbM1l_BoI
  # Other YouTube video ids
)

for i in "${array[@]}"
do
  if [ ! -f "$(pwd)/$i.mp3" ]; then
    docker run --rm -i -v "$(pwd)":/workdir:rw mikenye/youtube-dl $i --format "bestaudio" --audio-format mp3 --output "/workdir/%(id)s.%(ext)s"
    ffmpeg -i "$i.webm" -vn -ab 128k -ar 44100 -y "$i.mp3"; # TODO: Replace with docker container.
  else
    echo "File $i.mp3 exists, skipping."
  fi
done

ls

This requires docker and ffmpeg installed on the machine.

3. Uploaded the mp3 files on www/alarm/ folder. This will serve on https://myHassio/alarm/videoId.mp3
4. Then, I changed my Node-RED flow to first try using media_extractor; if that fails, I use the local folder with media_player.

This is a great workaround and was my primary path before using media_extractor but storing 8hour long music files is somewhat of a space hog especially when I use a randomize function to chose between different videos.

[minhphung1002]

Step 1: Find the location youtube_dl folder on your system with the command find / -type f -name 'YoutubeDL.py' Example Output : /var/lib/docker/overlay2/7f2ef3629a69c6b5f21d82c357c33de6cfcvvvxxxxxxx/merged/usr/local/lib/python3.10/site-packages/youtube_dl/ /var/lib/docker/overlay2/7f2ef3629a69c6b5f21d82c357c33de6cfcvvvxxxxxxx/diff/usr/local/lib/python3.10/site-packages/youtube_dl

Step 2: Download https://github.com/yt-dlp/yt-dlp/tree/master/yt_dlp Step 3: Replace all files and folders in the 2 folders you found in step 1 with the content you downloaded in step 2

[Scags104]

Step 1: Find the location youtube_dl folder on your system with the command find / -type f -name 'YoutubeDL.py' Example Output : /var/lib/docker/overlay2/7f2ef3629a69c6b5f21d82c357c33de6cfcvvvxxxxxxx/merged/usr/local/lib/python3.10/site-packages/youtube_dl/ /var/lib/docker/overlay2/7f2ef3629a69c6b5f21d82c357c33de6cfcvvvxxxxxxx/diff/usr/local/lib/python3.10/site-packages/youtube_dl

Step 2: Download https://github.com/yt-dlp/yt-dlp/tree/master/yt_dlp Step 3: Replace all files and folders in the 2 folders you found in step 1 with the content you downloaded in step 2

Thanks for this - any help porting this to non-docker installs? Using the HA OS. the find command doesnt work & /usr/lib/python3.10/site-packages doesnt have a youtube folder (usr/local/lib doesnt house python folder)

[anachronox99]

Thanks all. I was also inspired to find a workaround - but I just sourced my own audio file and used the GUI to play the media. Very impressed with how easy HA is making it to do things.

From the automation editor - add an action and pick 'Play Media'

And then pick your file from the filesystem and it will upload to the right place.

[Debugga-nb]

Step 1: Find the location youtube_dl folder on your system with the command find / -type f -name 'YoutubeDL.py' Example Output : /var/lib/docker/overlay2/7f2ef3629a69c6b5f21d82c357c33de6cfcvvvxxxxxxx/merged/usr/local/lib/python3.10/site-packages/youtube_dl/ /var/lib/docker/overlay2/7f2ef3629a69c6b5f21d82c357c33de6cfcvvvxxxxxxx/diff/usr/local/lib/python3.10/site-packages/youtube_dl

Step 2: Download https://github.com/yt-dlp/yt-dlp/tree/master/yt_dlp Step 3: Replace all files and folders in the 2 folders you found in step 1 with the content you downloaded in step 2

So this sounds like a dependency issue? How do we get your fix pushed to the master branch?

[lwsrbrts]

The way this integration is implemented relies on a reference to a published version of the youtube_dl python library. Currently the manifest.json has this line:

"requirements": ["youtube_dl==2021.12.17"]

Which means that it expects this version of this library which is a dependency. HA core does not maintain its own version of the youtube_dl library which is why it's referenced in this manner.

Since the bug exists in the upstream library (youtube_dl), that is the only place it can be fixed in order that HA gets the same fix.

What's complicating matters is that the library itself has been fixed (hence using the master branch of their repo to overwrite your HA copy makes it work) however, a new version of the library has not been released/published by the library owner and as such, the "newest" version remains as 2021.12.17.

The "fix" method discussed above messes with Docker's overlay2 storage driver. Each to their own but it's not a fix, more of a short-term workaround that probably won't survive updates to HA.

I imagine the only solution HA will implement is when youtube_dl releases a new version of the library that contains the actual code fix that's required and the HA side of that fix will involve little more than updating the requirements line of the manifest.json.

[Scags104]

Wonder if this could fix it as an intermediate solution before a new release is created. Points to a beta release created 3/18/23

See this post.

[Alexivia]

Also noticed this one is broken. Are the usual integration maintainers aware?

[lwsrbrts]

Also noticed this one is broken. Are the usual integration maintainers aware?

I'm sure they are but they can't do anything about it until the maintainers of youtube_dl release another version that fixes the issue. See my post a couple up as to why..

https://pypi.org/project/youtube_dl/

[skellycode]

For those who are inclined to do so, it's possible to create a custom component as a fork of media_extractor and adjust the code to utilise yt-dlp instead. This will give you a working version of media_extractor and allow you to customise the functionality as you see fit. I've done this myself (although with some workflow edits specific to my HA instance that makes it non-transferable) and it works a treat.

Since my exact setup probably won't work for others, here is a general process to help get you on the right track:

-1) Install yt-dlp within HA container (will need to repeat this step when updating HA - or add it to a build script): docker exec -it -u 0 homeassistant bash pip install yt-dlp -2) Edit import references in the media_extractor integration to be from yt-dlp instead of youtube_dl: from yt_dlp import YoutubeDL from yt_dlp.utils import DownloadError, ExtractorError -3) Tweak download function calls as needed (easy enough to debug in a standalone python script). The key part to edit is the get_stream_selector() function -4) Change component name in services.yaml and manifest.json to whatever you name your custom component -5) Add custom component to HA directory (as you would any other custom component) -6) Add config entry to enable custom component (in the same way you add media_extractor to your config).

Hope that helps for those who don't want to wait for an indeterminate amount of time for it to be fixed officially 😄

[lwsrbrts]

I like that idea as I'm starting to miss the option to use media extractor!

I think it would be good for the maintainers of the media_extractor integration to consider changing it to use yt-dlp instead as that seems to get far more regular updates. That said, I'm ignorant of the amount of work involved in doing that.

[kylehild]

I have created a pull request to upgrade to the new yt-dlp package instead of youtube-dl.