gallery-dl [macOS]ModuleNotFoundError: No module named 'yt_dlp','youtube

Without making any underlying changes to the python installation or system, gallery-dl can't call youtube_dl/yt_dlp for me since last update. Has this broken on macOS for anyone else on 1.26.1?

Oct 26 '23 20:10 glottisfaun0000

There haven't been any changes to ytdl code for several releases, so I couldn't tell you what's wrong without some more information.

You could check whether the python installation where gallery-dl is installed in and is running from also has youtube-dl/yt-dlp.

There was a recent macOS issue relating to a different ModuleNotFoundError. You say "Without making any underlying changes to the python installation or system", but maybe it's something similar: #4711

Oct 26 '23 20:10 mikf

I was worried that would be the case - this isn't even the first time I've had to troubleshoot this now that I think about it. I might have more python on macOS questions here than gallery-dl questions so maybe this should be closed I feel like

brew install python
brew install yt-dlp
brew install gallery-dl

should be compatible out-of-the-box.

which gallery-dl; which yt-dlp; which python
/opt/homebrew/bin/gallery-dl
/opt/homebrew/bin/yt-dlp
/opt/homebrew/opt/python@3/libexec/bin/python

I'm not sure how to check where the gallery-dl installation is apart from the which command. gallery-dl -vvv gives me

[downloader.ytdl][error] Cannot import module 'youtube_dl'
[downloader.ytdl][debug]
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/gallery-dl/1.26.1/libexec/lib/python3.12/site-packages/gallery_dl/ytdl.py", line 20, in import_module
    return __import__("yt_dlp")
           ^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'yt_dlp'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/gallery-dl/1.26.1/libexec/lib/python3.12/site-packages/gallery_dl/downloader/ytdl.py", line 44, in download
    module = ytdl.import_module(self.config("module"))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/gallery-dl/1.26.1/libexec/lib/python3.12/site-packages/gallery_dl/ytdl.py", line 22, in import_module
    return __import__("youtube_dl")
           ^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'youtube_dl'

edit: I notice that gallery-dl looks in /1.26.1/libexec/lib/python3.12, but if I do python --version I get Python 3.11.6 -- maybe that's the conflict? gallery-dl is in a python3.12 installation but yt-dlp is 3.11?

Oct 26 '23 21:10 glottisfaun0000

gallery-dl is in a python3.12 installation but yt-dlp is 3.11?

Probably something like this. I have no idea how exactly homebrew installs Python packages/programs, but it might be the case that each one gets installed into its own venv / has its own Python interpreter.

gallery-dl might use the environment in /1.26.1/libexec/lib/python3.12 while yt-dlp has its own, separate /VERSION/libexec/lib/python3.12 and both cannot see and import each other.

A solution would be to create a new venv (python -m venv gdl), activate it (. gdl/bin/activate), install both gallery-dl and yt-dlp in it (pip install gallery-dl yt-dlp), and run gallery-dl from there.

Maybe report this as a bug in the homebrew repo since this seems like a relatively recent change.

Oct 26 '23 21:10 mikf

Tried to venv method and got the same error unfortunately but thanks for the idea.

BUT I just got yt_dlp importing working based on the 3.11/3.12 difference by doing this extremely slapdash test.

brew installs yt_dlp the python module to /opt/homebrew/lib/python3.11/site-packages/yt_dlp

So I just did: ln -s /opt/homebrew/lib/python3.11/site-packages/yt_dlp /opt/homebrew/lib/python3.12/site-packages

to create a symlink, and gallery-dl is now importing it 😂 I really doubt this will survive updates, and also it seems that gallery-dl isn't using my config file specified by extractor.ytdl.config-file and downloader.ytdl.config-file, but hey, reddit videos are now downloading.

So there's a quick fix for macOS users who run into this- hoping someone who knows more about how brew works can come up with a real solution (I'm not sure what to report as an issue)

Oct 26 '23 21:10 glottisfaun0000

You were correct in https://github.com/mikf/gallery-dl/issues/4729#issuecomment-1781892599 with gallery-dl and yt-dlp using two different Python versions, it seems. This issue should resolve itself automatically when yt-dlp and others get updated to also use Python 3.12. No need to report anything, I just assumed it was perhaps an unintended mistake by homebrew packagers.

Oct 27 '23 12:10 mikf

As it turns out, the issues persists.

polaris@Umiko ~ % gallery-dl --verbose "https://old.reddit.com/user/fairy_tomie"
[gallery-dl][debug] Version 1.26.9
[gallery-dl][debug] Python 3.12.2 - macOS-14.4-x86_64-i386-64bit
[gallery-dl][debug] requests 2.31.0 - urllib3 2.2.1
[gallery-dl][debug] Configuration Files ['${XDG_CONFIG_HOME}/gallery-dl/config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://old.reddit.com/user/fairy_tomie'
[reddit][debug] Using RedditUserExtractor for 'https://old.reddit.com/user/fairy_tomie'
[cookies][debug] Extracting cookies from /Users/polaris/Library/Application Support/Firefox/Profiles/9wn2hwbi.dev-edition-default/cookies.sqlite
[cookies][info] Extracted 112 cookies from Firefox
[reddit][debug] Using default API credentials (client-id 6N9uN0krSDE-ig)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): oauth.reddit.com:443
[urllib3.connectionpool][debug] https://oauth.reddit.com:443 "GET /user/fairy_tomie/.json?limit=100&raw_json=1 HTTP/1.1" 200 21729
[reddit][debug] Using download archive '/Users/polaris/Library/Application Support/gallery-dl/archive.sqlite3'
[reddit][error] An unexpected error occurred: ModuleNotFoundError - No module named 'youtube_dl'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[reddit][debug]
Traceback (most recent call last):
  File "/usr/local/Cellar/gallery-dl/1.26.9/libexec/lib/python3.12/site-packages/gallery_dl/ytdl.py", line 20, in import_module
    return __import__("yt_dlp")
           ^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'yt_dlp'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/gallery-dl/1.26.9/libexec/lib/python3.12/site-packages/gallery_dl/job.py", line 128, in run
    self.dispatch(msg)
  File "/usr/local/Cellar/gallery-dl/1.26.9/libexec/lib/python3.12/site-packages/gallery_dl/job.py", line 182, in dispatch
    self.handle_queue(url, kwdict)
  File "/usr/local/Cellar/gallery-dl/1.26.9/libexec/lib/python3.12/site-packages/gallery_dl/job.py", line 364, in handle_queue
    extr = extractor.find(url)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/gallery-dl/1.26.9/libexec/lib/python3.12/site-packages/gallery_dl/extractor/__init__.py", line 207, in find
    return cls(match)
           ^^^^^^^^^^
  File "/usr/local/Cellar/gallery-dl/1.26.9/libexec/lib/python3.12/site-packages/gallery_dl/extractor/ytdl.py", line 26, in __init__
    ytdl_module = ytdl.import_module(config.get(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/gallery-dl/1.26.9/libexec/lib/python3.12/site-packages/gallery_dl/ytdl.py", line 22, in import_module
    return __import__("youtube_dl")
           ^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'youtube_dl'

And notice that both gallery-dl and yt-dlp are now at the same python version.

polaris@Umiko ~ % brew info gallery-dl
==> gallery-dl: stable 1.26.9 (bottled), HEAD
Command-line downloader for image-hosting site galleries and collections
https://github.com/mikf/gallery-dl
/usr/local/Cellar/gallery-dl/1.26.9 (346 files, 3MB) *
  Poured from bottle using the formulae.brew.sh API on 2024-03-23 at 22:49:41
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/g/gallery-dl.rb
License: GPL-2.0-only
==> Dependencies
Required: certifi ✔, [email protected] ✔
==> Options
--HEAD
        Install HEAD version

polaris@Umiko ~ % brew info yt-dlp
==> yt-dlp: stable 2024.03.10 (bottled), HEAD
Feature-rich command-line audio/video downloader
https://github.com/yt-dlp/yt-dlp
/usr/local/Cellar/yt-dlp/2024.03.10 (1,660 files, 16.6MB) *
  Built from source on 2024-03-18 at 23:56:04
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/y/yt-dlp.rb
License: Unlicense
==> Dependencies
Required: certifi ✔, [email protected] ✔
==> Options
--HEAD
        Install HEAD version
==> Caveats
zsh completions have been installed to:
  /usr/local/share/zsh/site-functions

I should also point out that the quick fix presented above didn't work as well, even after changing the path for the homebrew's intel directory.

Mar 25 '24 21:03 masterberg

This might not be the solution you want, but this works if you use pipx to install gallery-dl.

$ pipx install gallery-dl
$ pipx inject gallery-dl yt-dlp

The second commands installs yt-dlp as a dependency for gallery-dl

One liner:

pipx install gallery-dl && pipx inject gallery-dl yt-dlp

Apr 14 '24 02:04 EduardoRT

[macOS]ModuleNotFoundError: No module named 'yt_dlp','youtube_dl' since last update