meeb
Cannot Download - Expected outfile does not exist
The most recent download it made was 4 days ago. I believe that the ways to download YouTube videos have undergone recent changes. It is possible that YouTube has made some changes on their end. Anyone else experiencing this?
Downloading media for "xxx', attempted 14 times Error: "Failed to download media: JgZOnW728Qo (UUID: 4172c2fa-8653-46d8-b826-8d1451e6d236) to disk, expected outfile does not exist: /downloads/video/幻海航行/2018-11-10___JgZOnW728Qo_1080p-vp9-opus.mkv" Task will be retried at 2023-09-07 07:17:14
That error just means the expected outfile doesn't exist on disk. It's not the cause of the error. The error might be anything related to either YouTube, the downloading process with yt-dlp or your local system so anything from the media stream from YouTube isn't available to you don't have enough local disk space or it's on a SAN network share that doesn't support unicode characters.
There will be another error further up in the logs that contains the actual error.
I believe received
ERROR: [download] Got error: HTTP Error 403: Forbidden
Possible Tubesync downloader specifically blocked by YouTube? I can access YouTube via my laptop & smartphone within the same network.
2023/09/07 22:02:10 stderr Rescheduling Downloading media for "Fearless - Taylor Swift | Shania Yan Cover"
2023/09/07 22:02:10 stderr 2023-09-07 22:02:10,503 [tubesync/ERROR] Failed to download media: 1J5ikdy6P7g (UUID: f0bf6506-2053-436b-a9f9-c696838c1fb4) to disk, expected outfile does not exist: /downloads/video/ShaniaYan/2023-09-02_shania-yan_fearless-taylor-swift-shania-yan-cover_1J5ikdy6P7g_1080p-vp9-opus.mkv
2023/09/07 22:02:10 stderr ERROR: fragment 1 not found, unable to continue
2023/09/07 22:02:10 stderr ERROR: [download] Got error: HTTP Error 403: Forbidden
2023/09/07 22:02:10 stderr ERROR: fragment 1 not found, unable to continue
2023/09/07 22:02:10 stderr ERROR: [download] Got error: HTTP Error 403: Forbidden
2023/09/07 22:02:09 stderr 2023-09-07 22:02:09,458 [tubesync/INFO] Downloading media: 1J5ikdy6P7g (UUID: f0bf6506-2053-436b-a9f9-c696838c1fb4) to: "/downloads/video/ShaniaYan/2023-09-02_shania-yan_fearless-taylor-swift-shania-yan-cover_1J5ikdy6P7g_1080p-vp9-opus.mkv"
2023/09/07 22:00:40 stderr Rescheduling task Downloading media for "xxx" for 1:49:26 later at 2023-09-07 15:50:06.703964+00:00
2023/09/07 22:00:40 stderr common.errors.DownloadFailedException: Failed to download media: 8enaOMWZJtc (UUID: 8a65ebcd-702f-48b7-95b9-d4fa585fc1b8) to disk, expected outfile does not exist: /downloads/video/xxx/2023-09-07__90_8enaOMWZJtc_1080p-vp9-opus.mkv
2023/09/07 22:00:40 stderr raise DownloadFailedException(err)
2023/09/07 22:00:40 stderr File "/app/sync/tasks.py", line 407, in download_media
2023/09/07 22:00:40 stderr func(*args, **kwargs)
2023/09/07 22:00:40 stderr File "/usr/local/lib/python3.9/dist-packages/background_task/tasks.py", line 43, in bg_runner
2023/09/07 22:00:40 stderr Traceback (most recent call last):
2023/09/07 22:00:40 stderr Rescheduling Downloading media for "xxx"
2023/09/07 22:00:40 stderr 2023-09-07 22:00:40,665 [tubesync/ERROR] Failed to download media: 8enaOMWZJtc (UUID: 8a65ebcd-702f-48b7-95b9-d4fa585fc1b8) to disk, expected outfile does not exist: /downloads/video/xx/2023-09-07__90_8enaOMWZJtc_1080p-vp9-opus.mkv
2023/09/07 22:00:40 stderr ERROR: fragment 1 not found, unable to continue
2023/09/07 22:00:40 stderr ERROR: [download] Got error: HTTP Error 403: Forbidden
2023/09/07 22:00:40 stderr 2023-09-07 22:00:40,574 [tubesync/INFO] [youtube-dl] finished downloading: 2023-09-07__90_8enaOMWZJtc_1080p-vp9-opus.f248.webm - 10.00MiB in Unknown
2023/09/07 22:00:39 stderr 2023-09-07 22:00:39,171 [tubesync/INFO] Downloading media: 8enaOMWZJtc (UUID: 8a65ebcd-702f-48b7-95b9-d4fa585fc1b8) to: "/downloads/video/xxx/2023-09-07__90_8enaOMWZJtc_1080p-vp9-opus.mkv"
The video setting is set to fallback, never happen this before:
The actual error is "HTTP Error 403: Forbidden" which is YouTube has blocked you or your IP address. Does YouTube work for you in a browser off the same IP?
Ah, this may be that YouTube has marked the format tubesync is trying to download as "premium".
Thanks for figuring out the issue. Would it be possible to lower the quality option to fix it? Is there any other viable solution for this situation?
I probably need to see if I can filter out premium formats from being an index option as a patch to fix this.
I've ran into this as-well, the premium format can't be downloaded even after I provided cookies which are causing them to constantly fail. Not sure if theres a fix, I pay for premium so don't mind adding cookies but that doesn't seem to work when downloading the mkv
This started happening to me too. It started with one source and expanded from there. Now none of my sources are recording. I am shutting down my server until this resolved since there is need to continue indexing.
@psperry can you share your logs of the actual error you're encountering? "Expected outfile does not exist" is not the actual error. Is this related to your media formats being marked as premium?
@meeb -- Sorry to bother you. I'm an idiot. I pointed the app at the wrong storage that didn't have enough space. Slinking away now...
@meeb I'm experiencing the same issue, but I do have enough available storage space.
This is the page for one of the media pieces that fails out, I've had this same issue with around 4/450 media pieces.
Here's the files it leaves in the downloads directory after failing out:
And here's the logs from around the time those files left behind where left: https://pastebin.com/R00zcSW5
Let me know if any more info would be helpful. Thought I'd drop this in just in case it proved to be helpful in figuring this out. Thank you for the work you've done on this project, it's highly appreciated!
@Themis3000 from your linked logs:
[WARNING] Worker with pid 24813 was terminated due to signal 9
(repeated lots). Signal 9 is SIGKILL, something is killing our workers externally, as if it was being set a kill -9 [pid of worker]. Obviously if the download itself is forcibly killed then you'll have PART files (partial download) left on your drive as well as no expected completed outfile / download.
@Themis3000 from your linked logs:
[WARNING] Worker with pid 24813 was terminated due to signal 9(repeated lots). Signal 9 is SIGKILL, something is killing our workers externally, as if it was being set a
kill -9 [pid of worker]. Obviously if the download itself is forcibly killed then you'll have PART files (partial download) left on your drive as well as no expected completed outfile / download.
If I have time I'll have to test in another environment with the same config. I'm running on a Synology nas via the "Container Manager" application. I know they have some proprietary modifications to the base version of docker, perhaps that could be causing problems. Despite cpu & memory usage being low, it seems that the container hangs at times & fails to stop when I attempt to stop the container. Perhaps that part of the log is from me trying to stop the container while it seemed to be in an error state & the container not stopping correctly.
Looking further back in the log, it appears that I'm also getting the same HTTP 403 that @hoelee reported sometimes too. Looking at it further, I think this may just be a mirroring of the same issue @hoelee reported earlier.
Well, if you have 403's in addition to something on your NAS killing processes you're certainly in for an unreliable experience. Neither of those are really anything to do with tubesync itself though.
Well, if you have 403's in addition to something on your NAS killing processes you're certainly in for an unreliable experience. Neither of those are really anything to do with tubesync itself though.
I think that the processes being killed where initiated by me & I just didn't realize it because of how delayed it was. The only actual issue was the 403's, which confused me because I could visit and use youtube normally from the same ip address on my desktop.
Turns out the only issue was the image wasn't updated (facepalm). I've learned that Synology's container manager application will detect updates on images from ghcr.io & allow you to press an "update" button, but it'll silently fail in the background and the "update" button will disappear leaving the user thinking the container was successfully updated. I only realized when I connected to the container and checked the yt-dlp version (although I've now realized there's nifty version info at the bottom banner of the site, it would have saved me some trouble if I looked closer! It's even visible in my screenshot I uploaded & I didn't realize).
If anyone else finds this thread from google & is having similar troubles: just ssh into your nas and manually pull the image.
Everything is working as expected now, sorry to make you my rubber duck in the process.