meeb
Change matched media ad-hoc
Specifically in cases of failures but possibly more broadly. It would be nice to be able to change what format is being matched. This is in the rare cases where Youtube is advertising a format in the manifest that is not available.
@openmediavault:/srv/a3aac6c2-201d-4efb-9030-3a8fc20301af/Media/Youtube.tubesync# yt-dlp -f '302+140-dash' --merge-output-format mkv https://www.youtube.com/watch?v=PtjJE7zsGnY
[youtube] PtjJE7zsGnY: Downloading webpage
WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests
[youtube] PtjJE7zsGnY: Downloading android player API JSON
[youtube] PtjJE7zsGnY: Downloading iframe API JS
[youtube] PtjJE7zsGnY: Downloading player 10df06bb
[youtube] PtjJE7zsGnY: Downloading web player API JSON
[youtube] PtjJE7zsGnY: Downloading MPD manifest
[youtube] PtjJE7zsGnY: Downloading MPD manifest
[youtube] PtjJE7zsGnY: Downloading initial data API JSON
[info] PtjJE7zsGnY: Downloading 1 format(s): 302+140-dash
[dashsegments] Total fragments: 124
[download] Destination: Bethesda through 13 years of Evolution [PtjJE7zsGnY].f302.webm
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 1 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 2 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 3 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 4 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 5 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 6 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 7 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 8 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 9 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 10 of 10) ...
ERROR: Giving up after 10 fragment retries
ERROR: fragment 1 not found, unable to continue
[dashsegments] Total fragments: 64
[download] Destination: Bethesda through 13 years of Evolution [PtjJE7zsGnY].f140-dash.m4a
[download] 100% of 9.56MiB in 05:29
For media downloaded it would understand to purge the current media and redownload at the newly selected formats.
I believe I can get around this for now by downloading the media in a format that is available and using the import for the time being though? The import existing media instructions would likely be a workaround, yes? The instructions talk about pausing the downloading for the source. Is that required? Could I also edit the database directly to update the match instead and then let Tubesync handle the rest? I have failed to find the location for storing that within the current tables. I was expecting it within either background_tasks or sync_media but failed to find it.
Thanks for the issue. I've been looking into something like this on and off for a while. The issue really is how the yt-dlp and youtube-dl APIs are used and information they return. Currently, TubeSync just asks for the media to be downloaded, waits for the call to complete, then checks the expected output file exists. If it doesn't, it's marked as a failure and retried later. This works for things like connectivity issues, throttling, or other transient problems, but entirely fails for issues like this one where YouTube lies about the formats that are available which won't ever be resolved (probably).
Ideally like to do is fall back on formats that have permanent availability errors however this would require somehow extending this behavior so TubeSync can gain information on the type of failure and handle different failures with different results. Currently, the only way I've found is to search for the string HTTP Error 404 in log output, which is a pretty horrible way of detecting errors and why this has been a bit on the backburner to resolve. Ideally it would be nice if the youtube-dl APIs raised a specific exception that could be handled properly but this doesn't seem to be the case.
In the end though if there's no other work-around I suspect hacking it with log event chomping will get implemented somewhere...
As for importing existing media, it's advisable to pause then pause the source. This triggers a save signal and resets the appropriate tasks for the source. Without doing this it's possible to have tasks queued to download a video you've already manually imported and other messy overlaps. Yes you can relatively trivially update the database manually if you prefer, make sure the downloaded bool is set, then run reset-tasks on the command line which will pretty much do the same job as well.
You can see the logic of how the import existing media marks files as downloaded here:
https://github.com/meeb/tubesync/blob/main/tubesync/sync/management/commands/import-existing-media.py#L49
The pause / unpause logic stuff is just to keep the tasks in sync with the database state. Just running a reset-tasks CLI command will keep the tasks in sync after pretty much all database updates.
