Tasks stuck in Failed Download error loop at least once a day

[zpz5HAU-tgc3fgw2xwr]

I'm not sure why this would be happening, it doesn't seem to be consistent with any one channel or video. Basically at least once a day there are 1-2 videos that get stuck in a retry download loop and never get downloaded.

My solution to this has been a combination of deleting the .temp and .f___ files and restarting the container. Not sure if restarting is actually necessary, recently it seems to fix itself after just deleting the temp download files (although not every time, sometimes I have to delete those files 2-3 times)

If there's anything in my logs that I can send you to help debug this let me know and I'll check next time this happens. Currently have all my videos downloaded, but I'm sure I'll have this issue again tomorrow.

[meeb]

I've noticed this on a couple of videos myself, this may be due to yt-dlp doing something slightly differently to youtube-dl or that YouTube have changed something. Can you share a video URL that's failing for you as well as a copy/paste of the codec formats for the video (showing which formats have been matched / selected).

[zpz5HAU-tgc3fgw2xwr]

This one just started failing an hour ago

https://www.youtube.com/watch?v=4SoTx8yZk0k

Available Formats

ID: 139 , audio:mp4a.40.5 @48.796k / 22050Hz
ID: 249 , audio:opus @50.937k / 48000Hz
ID: 250 , audio:opus @64.754k / 48000Hz
ID: 140 , audio:mp4a.40.2 @129.48k / 44100Hz (matched)
ID: 251 , audio:opus @121.477k / 48000Hz
ID: 160 , 144p (256x136), fps:24, video:avc1.4d400c @49.134k
ID: 278 , 144p (256x136), fps:24, video:vp9 @58.535k
ID: 17 , 144p (176x144), fps:12, video:mp4v.20.3 @79.346k , audio:mp4a.40.2 @0.0k / 22050Hz
ID: 133 , 240p (426x224), fps:24, video:avc1.4d400d @101.591k
ID: 242 , 240p (426x224), fps:24, video:vp9 @89.369k
ID: 134 , 360p (640x338), fps:24, video:avc1.4d401e @190.427k
ID: 18 , 360p (640x338), fps:24, video:avc1.42001E @278.776k , audio:mp4a.40.2 @0.0k / 44100Hz
ID: 243 , 360p (640x338), fps:24, video:vp9 @154.523k
ID: 135 , 480p (854x450), fps:24, video:avc1.4d401e @374.507k
ID: 244 , 480p (854x450), fps:24, video:vp9 @250.543k
ID: 136 , 720p (1280x676), fps:24, video:avc1.64001f @689.453k
ID: 22 , 720p (1280x676), fps:24, video:avc1.64001F @817.905k , audio:mp4a.40.2 @0.0k / 44100Hz
ID: 247 , 720p (1280x676), fps:24, video:vp9 @437.875k
ID: 137 , 1080p (1920x1012), fps:24, video:avc1.640028 @1199.643k
ID: 248 , 1080p (1920x1012), fps:24, video:vp9 @818.305k
ID: 271 , 1440p (2560x1350), fps:24, video:vp9 @2102.285k
ID: 313 , 2160p (3840x2026), fps:24, video:vp9 @4327.83k (matched)

Matched formats

Combined: no match
Audio: 140 (exact match)
Video: 313 (fallback)

[meeb]

That seems to work fine for me. Can you drop into the TubeSync container and see if a manual download works? Something like:

$ docker exec -ti tubesync bash

then

$ yt-dlp -f '313+140' -o test.mkv https://www.youtube.com/watch\?v\=4SoTx8yZk0k

You can delete the test.mkv file once it's downloaded.

[zpz5HAU-tgc3fgw2xwr]

Yeah, manual downloads of the failing videos seems to work. I would expect as much, because I never have problems manually downloading with youtube-dl.

Maybe it has something to do with downloading newly uploaded videos...? And that's why when I finally get around to checking on it and deleting those bad temp files it ends up working on the next retry.

3/6 of the new videos from this morning had this problem.

Maybe a quick fix would be once a video has passed a certain number of consecutive fails, tubesync deletes all the temp files before retrying. I would say it should delete those files after 1 failure, but I think there's the off chance that you're downloading a 4hr video and your internet connection goes out, and those temp files are actually usable. But I'm not sure on that, and it's definitely not the case with most videos.

[meeb]

Yeah you could be right and it's a format availability issue. YouTube reports it as an available format, but it fails for a few hours until it's transcoded and published in more formats. This sort of intermittent weird problem is why there's about 6 retries for downloads. In theory the temporary files should be re-used or at least cleaned up once the download does actually succeed, but I would imagine there would be situations where the temporary files aren't cleaned up could occur so I'll have a look if what can be done to keep it tidy.

[zpz5HAU-tgc3fgw2xwr]

Dunno if you or someone else changed something, but I'd like to mention that for the first time in a while none of my videos have encountered this problem today. I'll report back if anything changes, but it's definitely a noticible improvement from over 50% of the new videos having this problem.

Oh - I also recently disabled TUBESYNC_WORKERS which used to be set to 4, so maybe that is related to why this was happening?

[meeb]

Nothing has been changed in TubeSync, so any changes would have been at YouTube directly.

The concurrency of workers wouldn't cause this, it was just fixed to 1 a while ago because lots of concurrent workers performing writes was problematic with the SQLite backend.

[zpz5HAU-tgc3fgw2xwr]

Nothing has been changed in TubeSync, so any changes would have been at YouTube directly.

The concurrency of workers wouldn't cause this, it was just fixed to 1 a while ago because lots of concurrent workers performing writes was problematic with the SQLite backend.

So yes, after some testing last week I found out the downloads were only ever failing 7+ times when I had TUBESYNC_WORKERS=4 set. After moving away from SQLite, this doesn't seem to be happening anymore, even with TUBESYNC_WORKERS=4 set.

I think a good fix for that would be if you're using SQLite as your backend, TubeSync ignores the TUBESYNC_WORKERS environment variable.

[meeb]

Shortly the TUBESYNC_WORKERS env var will be retired entirely so that'll also fix itself then.

[azerioxal]

I personally like having the ability to set TUBESYNC_WORKERS. It helps a lot with being able to index massive channels and multiple simultaneously. It hasn't caused any problems for me with several channels some of which have several thousand videos. I think you should consider leaving it in for advanced users.

[meeb]

There will be something that does a similar job that replaces it.

[azerioxal]

That's awesome.

Thanks for all the hard work you've done on this project.

[zpz5HAU-tgc3fgw2xwr]

Nothing has been changed in TubeSync, so any changes would have been at YouTube directly. The concurrency of workers wouldn't cause this, it was just fixed to 1 a while ago because lots of concurrent workers performing writes was problematic with the SQLite backend.

So yes, after some testing last week I found out the downloads were only ever failing 7+ times when I had TUBESYNC_WORKERS=4 set. After moving away from SQLite, this doesn't seem to be happening anymore, even with TUBESYNC_WORKERS=4 set.

I think a good fix for that would be if you're using SQLite as your backend, TubeSync ignores the TUBESYNC_WORKERS environment variable.

I spoke too soon, a couple of videos started having the same issue earlier this morning. Pretty weird, it was having such a good run too... I'm just going to assume it's still related to TUBESYNC_WORKERS and remove it while I'm not indexing new sources.