meeb
No Media to Index
I've been getting permanent failures with "No Media to Index" on existing entries (that have worked in the past) and new entries (which fail from the start).
Troubleshooting Steps:
- The keys are correct, as the generated tubesync url takes me to the channel.
- I use
:latest. - I've reset the tasks.
What else can we do to test?
Check what's in the logs. If there's an error indexing a channel you've added the reason for the failure will be in the container logs.
@meeb
2021-04-05 20:25:22,368 [tubesync/ERROR] ERROR: Unable to download JSON metadata: HTTP Error 410: Gone (caused by <HTTPError 410: 'Gone'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.,
Traceback (most recent call last):,
2021-04-05 20:25:27,833 [tubesync/ERROR] ERROR: Unable to download JSON metadata: HTTP Error 410: Gone (caused by <HTTPError 410: 'Gone'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.,
Rescheduling Index media from source "mymechanics",
File "/usr/local/lib/python3.7/dist-packages/background_task/tasks.py", line 43, in bg_runner,
func(*args, **kwargs),
File "/app/sync/tasks.py", line 163, in index_source_task,
raise NoMediaException(f'Source "{source}" (ID: {source_id}) returned no ',
common.errors.NoMediaException: Source "mymechanics" (ID: 68b9cbd8-b1f9-454c-9f27-a4be551853ea) returned no media to index, is the source key valid? Check the source configuration is correct and that the source is reachable,
Rescheduling task Index media from source "mymechanics" for 0:00:21 later at 2021-04-06 01:25:48.839286+00:00,
Rescheduling Index media from source "Whiteboard Finance",
common.errors.NoMediaException: Source "Whiteboard Finance" (ID: 49aa9afd-8758-4be8-ba5e-4d3a328ada36) returned no media to index, is the source key valid? Check the source configuration is correct and that the source is reachable,
Rescheduling task Index media from source "Whiteboard Finance" for 0:00:06 later at 2021-04-06 01:25:28.374148+00:00,
2021-04-05 20:25:27,479 [tubesync/DEBUG] [youtube:tab] mymechanics: Downloading webpage,
2021-04-05 20:25:27,738 [tubesync/DEBUG] [download] Downloading playlist: my mechanics - Videos,
2021-04-05 20:25:27,740 [tubesync/DEBUG] [youtube:tab] Downloading page 1,
2021-04-05 20:25:16,493 [tubesync/DEBUG] [youtube:tab] smartereveryday: Downloading webpage,
2021-04-05 20:25:16,743 [tubesync/DEBUG] [download] Downloading playlist: SmarterEveryDay - Videos,
2021-04-05 20:25:16,746 [tubesync/DEBUG] [youtube:tab] Downloading page 1,
2021-04-05 20:25:16,834 [tubesync/ERROR] ERROR: Unable to download JSON metadata: HTTP Error 410: Gone (caused by <HTTPError 410: 'Gone'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.,
Rescheduling Index media from source "smartereveryday",
common.errors.NoMediaException: Source "smartereveryday" (ID: a145039e-343a-44e1-a879-eb9848256e02) returned no media to index, is the source key valid? Check the source configuration is correct and that the source is reachable,
Rescheduling task Index media from source "smartereveryday" for 0:00:06 later at 2021-04-06 01:25:22.840418+00:00,
2021-04-05 20:25:17,227 [tubesync/DEBUG] [download] Downloading playlist: I Like To Make Stuff - Videos,
2021-04-05 20:25:16,990 [tubesync/DEBUG] [youtube:tab] UC6x7GwJxuoABSosgVXDYtTw: Downloading webpage,
2021-04-05 20:25:17,229 [tubesync/DEBUG] [youtube:tab] Downloading page 1,
2021-04-05 20:25:17,314 [tubesync/ERROR] ERROR: Unable to download JSON metadata: HTTP Error 410: Gone (caused by <HTTPError 410: 'Gone'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.,
Rescheduling Index media from source "I Like to Make Stuff",
common.errors.NoMediaException: Source "I Like to Make Stuff" (ID: eaaa505e-01c2-49c7-b1af-b5093ac0d686) returned no media to index, is the source key valid? Check the source configuration is correct and that the source is reachable,
2021-04-05 20:25:22,026 [tubesync/DEBUG] [youtube:tab] WhiteBoardFinance: Downloading webpage,
Rescheduling task Index media from source "I Like to Make Stuff" for 0:00:06 later at 2021-04-06 01:25:23.320379+00:00,
2021-04-05 20:25:22,281 [tubesync/DEBUG] [download] Downloading playlist: Marko - WhiteBoard Finance - Videos,
2021-04-05 20:25:22,283 [tubesync/DEBUG] [youtube:tab] Downloading page 1
Interesting, so the initial error is the "410 Gone" response from YouTube when attempting to download the index of video IDs. youtube-dl in TubeSync is up to date, so it might be a YouTube issue unpatched in youtube-dl or something else.
This is the channel you're trying to download?
https://www.youtube.com/c/mymechanics
If it is, it looks like this has a redirect from the vanity name to a usual channel ID. Can you try changing the "key" of the source to "UCMrMVIBtqFW6O0-MWq26gqw" and reset your tasks again to see if that helps?
If not, I'll poke about further.
For me, using the command like test tool that channel does index videos properly:
$ ./manage.py youtube-dl-info https://www.youtube.com/c/mymechanics/videos
Showing information for URL: https://www.youtube.com/c/mymechanics/videos
2021-04-06 01:35:09,942 [tubesync/DEBUG] [youtube:tab] mymechanics: Downloading webpage
2021-04-06 01:35:10,324 [tubesync/DEBUG] [download] Downloading playlist: my mechanics - Videos
2021-04-06 01:35:10,325 [tubesync/DEBUG] [youtube:tab] Downloading page 1
2021-04-06 01:35:10,628 [tubesync/DEBUG] [youtube:tab] playlist my mechanics - Videos: Downloading 35 videos
2021-04-06 01:35:10,629 [tubesync/DEBUG] [download] Downloading video 1 of 35
2021-04-06 01:35:10,644 [tubesync/DEBUG] {"_type": "url_transparent", "ie_key": "Youtube", "id": "2CHXBMwxRWA", "url": "2CHXBMwxRWA", "title": "Antique Rusty Cleaver - Restoration", "description": null, "duration": null, "view_count": 4773409, "uploader": null}
[... truncated videos here etc ...]
Changed channel to "by ID" and set the key to the one you provided:
2021-04-05 20:41:04,319 [tubesync/DEBUG] [youtube:tab] UCMrMVIBtqFW6O0-MWq26gqw: Downloading webpage,
2021-04-05 20:41:04,576 [tubesync/DEBUG] [download] Downloading playlist: my mechanics - Videos,
2021-04-05 20:41:04,578 [tubesync/DEBUG] [youtube:tab] Downloading page 1,
2021-04-05 20:41:04,671 [tubesync/ERROR] ERROR: Unable to download JSON metadata: HTTP Error 410: Gone (caused by <HTTPError 410: 'Gone'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.,
Rescheduling Index media from source "mymechanics",
Traceback (most recent call last):,
File "/usr/local/lib/python3.7/dist-packages/background_task/tasks.py", line 43, in bg_runner,
File "/app/sync/tasks.py", line 163, in index_source_task,
func(*args, **kwargs),
raise NoMediaException(f'Source "{source}" (ID: {source_id}) returned no ',
common.errors.NoMediaException: Source "mymechanics" (ID: 68b9cbd8-b1f9-454c-9f27-a4be551853ea) returned no media to index, is the source key valid? Check the source configuration is correct and that the source is reachable,
Rescheduling task Index media from source "mymechanics" for 0:00:21 later at 2021-04-06 01:41:25.676456+00:00
OK, does the URL https://www.youtube.com/c/mymechanics/videos work for you in a browser assuming you're on the same IP address as your TubeSync install?
My tubesync instance is behind a vpn. My current machine is also behind the same VPN (in a different location, though, as it's randomized). But it's the same provider, so i would assume it would be the same. And yes, my current machine can resolve that url and it loads just fine.
If there was a blocking/throttling issue with YouTube it would probably be specifically IP-based so to test that properly you would really need to be on the same IP exactly. I'll test this further when I get a moment though as this is either an issue with the channel setup and how it's parsed in TubeSync (with a redirect or two possibly not being followed and sub-channels expanded) or an issue with youtube-dl that's not patched yet so will require some time to poke further into.
@meeb I'm still getting this error and haven't been able to use tubesync in a long time. Any ideas to continue investigating?
@meeb I temporarily launched this on another VM, which is not behind a vpn. It's working there. So I have a feeling something needs to be configured to allow for this kind of setup (the VM behind the vpn is using wireguard, if that helps). I'd be glad to help troubleshoot and get on a session with you.
Using a VPN would be purely a hosting issue (especially a layer3 VPN like wireguard), there would be nothing to configure in TubeSync or any other container to use a VPN. It really does sound like whatever IP you were using on the VPN is throttled or blocked by YouTube.
That's what I thought, but I manually exec'd into the container and ran YouTube-dl and it worked fine.
There appears to be different throttling at YouTube for downloading a single video and indexing a load of videos. You are likely still just getting throttled. Try deleting your sources and add in a single, small channel and see if that works as a test.