tubesync icon indicating copy to clipboard operation
tubesync copied to clipboard

Published 20 hours ago • verified publisher icon meeb

TubeSync skipping all downloads

Open martinkeat opened this issue 11 months ago • 23 comments

Tubesync 14.3 in Docker, logs show the following. All videos from all sources (set to download 1080p - or next best format/codec - VP9, Opus)

2024-02-29 08:55:56,896 [tubesync/WARNING] Download task triggered for media: 1yfuCpcX7EU (UUID: 7a037a27-498e-44fa-a6b9-a9a06d6f2c0f) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:56,930 [tubesync/WARNING] Download task triggered for media: q98vhT0QhFM (UUID: 83a06363-b801-4301-ada6-82a1c6238003) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,032 [tubesync/WARNING] Download task triggered for media: 3FeserOHtgg (UUID: b1613aba-e13c-4678-a580-03c36e920a05) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,089 [tubesync/WARNING] Download task triggered for media: Gjzhi6aHU74 (UUID: 0d72f1d9-4e05-406c-974a-e32465cf74d5) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,150 [tubesync/WARNING] Download task triggered for media: lloVQPM8TOY (UUID: c81e9b58-3e44-4919-9658-821298f71c00) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,209 [tubesync/WARNING] Download task triggered for media: dqo2DxUBb-o (UUID: 6451a90c-75d6-40ab-b724-a13baed9e376) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,249 [tubesync/WARNING] Download task triggered for media: 05S6p-HkrtY (UUID: 80c52d99-4871-47f3-918a-a14cc0bcfbec) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,320 [tubesync/WARNING] Download task triggered for media: lC1UG_hrJeo (UUID: 6ad21505-0a39-4971-b489-61550cd1b4fe) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,362 [tubesync/WARNING] Download task triggered for media: 5wMQNIK9OIk (UUID: fb47e570-7495-4349-aa0c-22d506bb2469) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,448 [tubesync/WARNING] Download task triggered for media: rKHgSM-IK3s (UUID: 4a0b40b6-5a63-4cee-827c-d07487d89628) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,476 [tubesync/WARNING] Download task triggered for media: Xvz6MEOtwyo (UUID: 47adc696-0be4-41b6-b04b-d35d21024293) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,514 [tubesync/WARNING] Download task triggered for media: s0RK_LomzWg (UUID: 81feace4-08d1-4285-9800-19d9983517d6) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,578 [tubesync/WARNING] Download task triggered for media: TaD3SgPuccE (UUID: 865f6d4a-f6fa-4aa6-8801-bace69cdbb58) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,584 [tubesync/WARNING] Download task triggered for media: 15WDBuvovXo (UUID: 40796111-378b-4367-93d7-1eab2c3ab05b) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,687 [tubesync/WARNING] Download task triggered for media: r0yI78cFeVs (UUID: 2e9b26c3-5987-44a9-a698-c8b7e6c220a1) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,742 [tubesync/WARNING] Download task triggered for media: PB4MXzB-GZ4 (UUID: fcb9b84a-fbeb-46ba-b838-99d133aa7535) but it is now marked to be skipped, not downloading thumbnail 2024-02-29 08:55:57,842 [tubesync/WARNING] Download task triggered for media: Sw_jVHpGcjg (UUID: 98c5e7c7-2be7-4ca3-9333-ab503e22d30b) but it is now marked to be skipped, not downloading thumbnail

martinkeat avatar Feb 29 '24 08:02 martinkeat

Settings for one of the sources:

`

Fallback | Get next best resolution or codec instead -- | -- Skipping? |   Can download? |   Available formats | ID: sb3ID: sb2ID: sb1ID: sb0ID: 233ID: 234ID: 599 , audio:mp4a.40.5 @30.819k / 22050HzID: 600 , audio:opus @32.293k / 48000HzID: 139 , audio:mp4a.40.5 @48.816k / 22050HzID: 249 , audio:opus @47.248k / 48000HzID: 250 , audio:opus @61.261k / 48000HzID: 140 , audio:mp4a.40.2 @129.505k / 44100HzID: 251 , audio:opus @117.514k / 48000Hz (matched)ID: 597 , 144p (188x144), fps:13, video:avc1.4d400b @24.903kID: 602 , (188x144), fps:13.0, video:vp09.00.10.08 @74.014kID: 598 , 144p (188x144), fps:13, video:vp9 @17.875kID: 269 , (188x144), fps:25.0, video:avc1.4D400B @138.26kID: 160 , 144p (188x144), fps:25, video:avc1.4D400B @35.954kID: 603 , (188x144), fps:25.0, video:vp09.00.11.08 @153.937kID: 278 , 144p (188x144), fps:25, video:vp09.00.11.08 @64.746kID: 229 , (312x240), fps:25.0, video:avc1.4D400D @241.054kID: 133 , 240p (312x240), fps:25, video:avc1.4D400D @73.429kID: 604 , (312x240), fps:25.0, video:vp09.00.20.08 @251.379kID: 242 , 240p (312x240), fps:25, video:vp09.00.20.08 @67.968kID: 230 , (468x360), fps:25.0, video:avc1.4D4015 @622.653kID: 134 , 360p (468x360), fps:25, video:avc1.4D4015 @153.527kID: 18 , 360p (468x360), fps:25, video:avc1.42001E @282.633k , audio:mp4a.40.2 @Nonek / 44100HzID: 605 , (468x360), fps:25.0, video:vp09.00.21.08 @586.728kID: 243 , 360p (468x360), fps:25, video:vp09.00.21.08 @117.215kID: 231 , (626x480), fps:25.0, video:avc1.4D401E @1027.605kID: 135 , 480p (626x480), fps:25, video:avc1.4D401E @285.895kID: 606 , (626x480), fps:25.0, video:vp09.00.30.08 @700.298kID: 244 , 480p (626x480), fps:25, video:vp09.00.30.08 @163.235k (matched) Matched formats | Combined: no matchAudio: 251 (exact match)Video: 244 (fallback) `

martinkeat avatar Feb 29 '24 09:02 martinkeat

Can you screenshot the status for one of the media items that are skipped? E.g. /media/7a037a27-498e-44fa-a6b9-a9a06d6f2c0f

meeb avatar Feb 29 '24 09:02 meeb

image image image

martinkeat avatar Feb 29 '24 19:02 martinkeat

image image image

martinkeat avatar Feb 29 '24 19:02 martinkeat

Your media items have failed to download enough times that they have been marked as failed and skipped. Make sure you've updated tubesync to the latest release and try using the "unskip" button on a single item of currently skipped media and see if that works.

As to why the downloads originally failed, no idea this could be because tubesync needed an update or a network connection issue or disk was full or YouTube limited your IP etc. etc.

meeb avatar Mar 01 '24 11:03 meeb

Hi @meeb

I actually googled the issue first before coming here. Manual skipping and unskipping does not resolve the issue. I tried a number from each source. Also, the version is the latest version and is set to update automatically each night.

The IP address might be something though. I'll check that and get back to you.

martinkeat avatar Mar 01 '24 13:03 martinkeat

You can test yt-dlp directly, drop into the tubesync container with:

$ docker exec -ti tubesync bash

Then you can test it with:

$ cd /downloads
$ yt-dlp "https://www.youtube.com/watch?v=1yfuCpcX7EU"

If that errors then it's your connection or some other global issue.

meeb avatar Mar 01 '24 14:03 meeb

Hi @meeb - sorry for the late reply. Ran that. It did not error. So not what we thought. Output from the terminal is below.

FYI, I am running it on unraid so didn't need to docker exec to drop into the container.

#  cd /downloads
# yt-dlp "https://www.youtube.com/watch?v=1yfuCpcX7EU"
[youtube] Extracting URL: https://www.youtube.com/watch?v=1yfuCpcX7EU
[youtube] 1yfuCpcX7EU: Downloading webpage
[youtube] 1yfuCpcX7EU: Downloading ios player API JSON
[youtube] 1yfuCpcX7EU: Downloading android player API JSON
[youtube] 1yfuCpcX7EU: Downloading m3u8 information
[info] 1yfuCpcX7EU: Downloading 1 format(s): 616+251
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 56
[download] Destination: Sade - Your Love Is King (Live 2011) [1yfuCpcX7EU].f616.mp4
[download] 100% of   86.85MiB in 00:00:27 at 3.19MiB/s
[download] Destination: Sade - Your Love Is King (Live 2011) [1yfuCpcX7EU].f251.webm
[download] 100% of    4.54MiB in 00:00:00 at 7.28MiB/s
[Merger] Merging formats into "Sade - Your Love Is King (Live 2011) [1yfuCpcX7EU].webm"
Deleting original file Sade - Your Love Is King (Live 2011) [1yfuCpcX7EU].f616.mp4 (pass -k to keep)
Deleting original file Sade - Your Love Is King (Live 2011) [1yfuCpcX7EU].f251.webm (pass -k to keep)
#

martinkeat avatar Mar 03 '24 08:03 martinkeat

When you unskip a skipped video are there any errors on your tasks page?

meeb avatar Mar 03 '24 09:03 meeb

@meeb Yes.

image

martinkeat avatar Mar 03 '24 09:03 martinkeat

That's the media item page not the tasks page.

meeb avatar Mar 03 '24 09:03 meeb

Sorry. Wrong screenshot

image

martinkeat avatar Mar 03 '24 09:03 martinkeat

Did you specifically unskip that YouTube item? jDvhwHSnUJQ? "No valid format available" means, assuming you don't have crazy source requirements, that the metadata is missing or invalid. Anything on the container logs themselves?

meeb avatar Mar 03 '24 09:03 meeb

Yup. I did it there and then. Hence why you see the "downloading thumbnail" directly below the error.

Source requirements aren't out there. Standard VP9 with OPUS 1080p set to get the next available quality or codec. Thats pretty much it.

martinkeat avatar Mar 03 '24 09:03 martinkeat

Log:

2024-03-03 09:19:17,205 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:01:26 later at 2024-03-03 09:20:43.212297+00:00
Unauthorized: /
2024-03-03 09:20:47,470 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:04:21 later at 2024-03-03 09:25:08.477666+00:00
10.0.10.11 - davie [03/Mar/2024:09:19:04 +0000] "GET / HTTP/1.1" 200 2395 "https://tubesync.methven.app/media" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:05 +0000] "GET /tasks HTTP/1.1" 200 4825 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:10 +0000] "GET /tasks HTTP/1.1" 200 3405 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:13 +0000] "GET /tasks HTTP/1.1" 200 2773 "https://tubesync.methven.app/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:19 +0000] "GET /source/f7dda3c8-0cbe-49b8-971a-4c998c583a47 HTTP/1.1" 200 3378 "https://tubesync.methven.app/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:21 +0000] "GET / HTTP/1.1" 200 2395 "https://tubesync.methven.app/source/f7dda3c8-0cbe-49b8-971a-4c998c583a47" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - - [03/Mar/2024:09:20:02 +0000] "GET / HTTP/1.1" 401 120 "-" "-"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /sources HTTP/1.1" 200 2449 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /static/styles/tubesync.css HTTP/1.1" 200 34554 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /static/images/favicon.ico HTTP/1.1" 304 0 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/fontawesome/fa-regular-400.woff2 HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/roboto/roboto-regular.woff HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/roboto/roboto-bold.woff HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"

martinkeat avatar Mar 03 '24 09:03 martinkeat

Tried a different one:

sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:04:21 later at 2024-03-03 09:25:08.477666+00:00
10.0.10.11 - davie [03/Mar/2024:09:19:04 +0000] "GET / HTTP/1.1" 200 2395 "https://tubesync.methven.app/media" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:05 +0000] "GET /tasks HTTP/1.1" 200 4825 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:10 +0000] "GET /tasks HTTP/1.1" 200 3405 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:13 +0000] "GET /tasks HTTP/1.1" 200 2773 "https://tubesync.methven.app/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:19 +0000] "GET /source/f7dda3c8-0cbe-49b8-971a-4c998c583a47 HTTP/1.1" 200 3378 "https://tubesync.methven.app/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:21 +0000] "GET / HTTP/1.1" 200 2395 "https://tubesync.methven.app/source/f7dda3c8-0cbe-49b8-971a-4c998c583a47" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - - [03/Mar/2024:09:20:02 +0000] "GET / HTTP/1.1" 401 120 "-" "-"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /sources HTTP/1.1" 200 2449 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /static/styles/tubesync.css HTTP/1.1" 200 34554 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /static/images/favicon.ico HTTP/1.1" 304 0 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/fontawesome/fa-regular-400.woff2 HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/roboto/roboto-regular.woff HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/roboto/roboto-bold.woff HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
Unauthorized: /
10.0.10.11 - - [03/Mar/2024:09:22:30 +0000] "GET / HTTP/1.1" 401 120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
Unauthorized: /tasks
10.0.10.11 - - [03/Mar/2024:09:22:30 +0000] "GET /tasks HTTP/1.1" 401 120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
10.0.10.11 - - [03/Mar/2024:09:22:30 +0000] "GET /static/styles/tubesync.css HTTP/1.1" 200 34554 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
Unauthorized: /source/f7dda3c8-0cbe-49b8-971a-4c998c583a47
10.0.10.11 - - [03/Mar/2024:09:22:31 +0000] "GET /source/f7dda3c8-0cbe-49b8-971a-4c998c583a47 HTTP/1.1" 401 120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
Unauthorized: /sources
10.0.10.11 - - [03/Mar/2024:09:22:31 +0000] "GET /sources HTTP/1.1" 401 120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:22:52 +0000] "GET /source/f7dda3c8-0cbe-49b8-971a-4c998c583a47 HTTP/1.1" 200 3378 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:22:52 +0000] "GET /static/fonts/fontawesome/fa-solid-900.woff2 HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:22:53 +0000] "GET /static/fonts/fontawesome/fa-brands-400.woff2 HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:22:55 +0000] "GET /tasks HTTP/1.1" 200 2773 "https://tubesync.methven.app/source/f7dda3c8-0cbe-49b8-971a-4c998c583a47" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:22:57 +0000] "GET /sources HTTP/1.1" 200 2449 "https://tubesync.methven.app/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:00 +0000] "GET /media HTTP/1.1" 200 2113 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:01 +0000] "GET /static/images/nothumb.png HTTP/1.1" 304 0 "https://tubesync.methven.app/media" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:02 +0000] "GET /media?show_skipped=yes HTTP/1.1" 200 10490 "https://tubesync.methven.app/media" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:09 +0000] "GET /media/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 200 3658 "https://tubesync.methven.app/media?show_skipped=yes" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:16 +0000] "GET /media-skip/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 200 2222 "https://tubesync.methven.app/media/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:17 +0000] "POST /media-skip/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 302 0 "https://tubesync.methven.app/media-skip/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:17 +0000] "GET /media/df6438a5-240c-4eb3-a53f-0004c88dfdec?message=skipped HTTP/1.1" 200 2671 "https://tubesync.methven.app/media-skip/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:20 +0000] "GET /media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 200 2179 "https://tubesync.methven.app/media/df6438a5-240c-4eb3-a53f-0004c88dfdec?message=skipped" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:21 +0000] "GET /static/styles/tubesync.css HTTP/1.1" 200 34554 "https://tubesync.methven.app/media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:21 +0000] "GET /static/images/favicon.ico HTTP/1.1" 304 0 "https://tubesync.methven.app/media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:23:24,229 [tubesync/INFO] Scheduling task to download metadata for: https://www.youtube.com/watch?v=foIjI685ino
10.0.10.11 - davie [03/Mar/2024:09:23:24 +0000] "POST /media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 302 0 "https://tubesync.methven.app/media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:24 +0000] "GET /media/df6438a5-240c-4eb3-a53f-0004c88dfdec?message=enabled HTTP/1.1" 200 2727 "https://tubesync.methven.app/media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:23:27,750 [tubesync/INFO] Downloading media: foIjI685ino (UUID: df6438a5-240c-4eb3-a53f-0004c88dfdec) to: "/downloads/video/Sade/..mkv"
Rescheduling Downloading media for "foIjI685ino"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 410, in download_media
    format_str, container = media.download_media()
                            ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1405, in download_media
    raise NoFormatException(f'Cannot download, media "{self.pk}" ({self}) has '
common.errors.NoFormatException: Cannot download, media "df6438a5-240c-4eb3-a53f-0004c88dfdec" (foIjI685ino) has no valid format available
Rescheduling task Downloading media for "foIjI685ino" for 0:00:06 later at 2024-03-03 09:23:33.755498+00:00
10.0.10.11 - davie [03/Mar/2024:09:23:29 +0000] "GET /media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 200 2176 "https://tubesync.methven.app/media/df6438a5-240c-4eb3-a53f-0004c88dfdec?message=skipped" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:23:29,894 [tubesync/WARNING] Media: Sade / foIjI685ino is older than 2 days, skipping
2024-03-03 09:23:29,903 [tubesync/INFO] Scheduling task to download thumbnail for: Sade - Cherish The Day (Lovers Live) from: https://i.ytimg.com/vi/foIjI685ino/maxresdefault.jpg
2024-03-03 09:23:29,910 [tubesync/INFO] Saved 81205 bytes of metadata for: Sade / df6438a5-240c-4eb3-a53f-0004c88dfdec
10.0.10.11 - davie [03/Mar/2024:09:23:31 +0000] "GET /media-skip/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 200 2227 "https://tubesync.methven.app/media/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:23:32,856 [tubesync/WARNING] Download task triggered for media: foIjI685ino (UUID: df6438a5-240c-4eb3-a53f-0004c88dfdec) but it is now marked to be skipped, not downloading thumbnail
2024-03-03 09:23:37,887 [tubesync/WARNING] Download task triggered for  media: foIjI685ino (UUID: df6438a5-240c-4eb3-a53f-0004c88dfdec) but it is now marked to be skipped, not downloading

martinkeat avatar Mar 03 '24 09:03 martinkeat 

2024-03-03 09:23:37,887 [tubesync/WARNING] Download task triggered for  media: foIjI685ino (UUID: df6438a5-240c-4eb3-a53f-0004c88dfdec) but it is now marked to be skipped, not downloading
Unauthorized: /media/df6438a5-240c-4eb3-a53f-0004c88dfdec
10.0.10.11 - - [03/Mar/2024:09:24:55 +0000] "GET /media/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 401 120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
2024-03-03 09:25:09,616 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:10:30 later at 2024-03-03 09:35:39.621931+00:00

martinkeat avatar Mar 03 '24 09:03 martinkeat

Well, I have zero idea why command line yt-dlp works but calling it over the API doesn't but the reason your downloads are failing is because they're all marked as claimed under the YouTube copyright claimed:

2024-03-03 09:25:09,616 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy

meeb avatar Mar 03 '24 09:03 meeb

Dunno if it is relevant but I just tried deleting and re-adding the source: https://www.youtube.com/@SadeSME

And noticed that it seems to be inserting a /c/ in the URL for the channel.

image

martinkeat avatar Mar 03 '24 09:03 martinkeat

Yeah you need to convert that @ handle to a channel ID using something like:

https://www.streamweasels.com/tools/youtube-channel-id-and-user-id-convertor/

For example SadeSME is UCegKDBFMUSQNeWe4FAB7aig then add a source by channel ID and use UCegKDBFMUSQNeWe4FAB7aig as the ID.

But, that won't make the media items which are failing to download due to copyright claims magically work.

meeb avatar Mar 03 '24 09:03 meeb

Just one problem with that, you'll note the name of the person as sarah something or other. Well if I disable/enable love thy neighbour, which has nothing to do with sade, and is also way outside the fair use for the UK, it is still supposedly copyrighted by the same person:

 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:33:40 +0000] "GET /media-skip/ab04eb31-cd13-4260-bce1-98bf1f2888f2 HTTP/1.1" 200 2229 "https://tubesync.methven.app/media/ab04eb31-cd13-4260-bce1-98bf1f2888f2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:33:43 +0000] "POST /media-skip/ab04eb31-cd13-4260-bce1-98bf1f2888f2 HTTP/1.1" 302 0 "https://tubesync.methven.app/media-skip/ab04eb31-cd13-4260-bce1-98bf1f2888f2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:33:43 +0000] "GET /media/ab04eb31-cd13-4260-bce1-98bf1f2888f2?message=skipped HTTP/1.1" 200 2698 "https://tubesync.methven.app/media-skip/ab04eb31-cd13-4260-bce1-98bf1f2888f2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:33:56 +0000] "GET /media-enable/ab04eb31-cd13-4260-bce1-98bf1f2888f2 HTTP/1.1" 200 2177 "https://tubesync.methven.app/media/ab04eb31-cd13-4260-bce1-98bf1f2888f2?message=skipped" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:33:59,128 [tubesync/INFO] Scheduling task to download metadata for: https://www.youtube.com/watch?v=Ik1C0aYvqVk
10.0.10.11 - davie [03/Mar/2024:09:33:59 +0000] "POST /media-enable/ab04eb31-cd13-4260-bce1-98bf1f2888f2 HTTP/1.1" 302 0 "https://tubesync.methven.app/media-enable/ab04eb31-cd13-4260-bce1-98bf1f2888f2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:33:59 +0000] "GET /media/ab04eb31-cd13-4260-bce1-98bf1f2888f2?message=enabled HTTP/1.1" 200 2716 "https://tubesync.methven.app/media-enable/ab04eb31-cd13-4260-bce1-98bf1f2888f2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
**2024-03-03 09:34:01,855 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy**
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:00:06 later at 2024-03-03 09:34:08.681956+00:00
2024-03-03 09:34:15,386 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:00:21 later at 2024-03-03 09:34:36.552793+00:00
10.0.10.11 - davie [03/Mar/2024:09:34:28 +0000] "GET /media HTTP/1.1" 200 2113 "https://tubesync.methven.app/media/ab04eb31-cd13-4260-bce1-98bf1f2888f2?message=enabled" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:34:43,634 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:01:26 later at 2024-03-03 09:36:10.336515+00:00
10.0.10.11 - davie [03/Mar/2024:09:34:46 +0000] "GET /media?show_skipped=yes HTTP/1.1" 200 7094 "https://tubesync.methven.app/media" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"

Yeah you need to convert that @ handle to a channel ID using something like:

https://www.streamweasels.com/tools/youtube-channel-id-and-user-id-convertor/

For example SadeSME is UCegKDBFMUSQNeWe4FAB7aig then add a source by channel ID and use UCegKDBFMUSQNeWe4FAB7aig as the ID.

But, that won't make the media items which are failing to download due to copyright claims magically work.

martinkeat avatar Mar 03 '24 09:03 martinkeat

Yeah, they all have the same error in your logs:

2024-03-03 09:34:01,855 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy**

There's obviously not much tubesync can do about YouTube returning that error.

I also get that error with that video:

$ yt-dlp "https://www.youtube.com/watch?v=Ik1C0aYvqVk"
[youtube] Extracting URL: https://www.youtube.com/watch?v=Ik1C0aYvqVk
[youtube] Ik1C0aYvqVk: Downloading webpage
[youtube] Ik1C0aYvqVk: Downloading ios player API JSON
[youtube] Ik1C0aYvqVk: Downloading android player API JSON
ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy

Basically your error is because the videos aren't available on YouTube.

meeb avatar Mar 03 '24 09:03 meeb

Oh and if your "love thy neighbour" tool works like a VPN it's entirely possible the copyright claim is regional and might not be present in some countries which is why that might impact it.

meeb avatar Mar 03 '24 11:03 meeb