Issue with SponserBlock / What is blocked.

[Lucas319315]

I check my sources menu and when I go into it shows sponser block enabled but the "what blocked?" is empty. I go and re-add the categories again but when I eventually go back to it that section is blank again.

[meeb]

Thanks for the issue, I'll look into it.

[Lucas319315]

Thanks for the issue, I'll look into it.

Sounds good! Good luck trying to resolve this.

[Snijder]

This is an issue for me too, sponsorblock also does not seem to do anything. I have this line in the logs: ERROR: Object of type CommaSepChoiceField is not JSON serializable which might be related.

[Snijder]

I managed to hack it back into working order by manually editing the youtube.py and copy pasting from a code snippet in a reddit post, I don't know enough about python to submit a PR unfortunately.

'postprocessors': [
    {'key': 'SponsorBlock'},
    {'key': 'ModifyChapters', 'remove_sponsor_segments': ['sponsor', 'intro', 'outro', 'selfpromo', 'preview', 'filler', 'interaction']}
 ]

Removing https://github.com/meeb/tubesync/blob/main/tubesync/sync/youtube.py#L117 also got rid of the error I mentioned in my previous post. (I appended ModifyChapters below it.)

(Not completely sure if this would also solve the settings disappearing)

[meeb]

Your patch is pretty much exactly what the default sponsor block management code should do, selecting the tick boxes should just generate the list like ['sponsor', 'intro', 'outro', 'selfpromo', 'preview', 'filler', 'interaction'] depending on your selection automatically. A PR containing this would override the web UI.

There's a bug somewhere in the handling and creation of the the list that only affects some deployments so it's tricky to catch.

[mylogon341]

fyi im getting this issue also. I can give details of my setup /the yt channel if you think it'd help with your debugging/ resolution of this.

thanks.

[Lucas319315]

I managed to hack it back into working order by manually editing the youtube.py and copy pasting from a code snippet in a reddit post, I don't know enough about python to submit a PR unfortunately.

'postprocessors': [
    {'key': 'SponsorBlock'},
    {'key': 'ModifyChapters', 'remove_sponsor_segments': ['sponsor', 'intro', 'outro', 'selfpromo', 'preview', 'filler', 'interaction']}
 ]

Removing https://github.com/meeb/tubesync/blob/main/tubesync/sync/youtube.py#L117 also got rid of the error I mentioned in my previous post. (I appended ModifyChapters below it.)

(Not completely sure if this would also solve the settings disappearing)

I am using this in unraid do you know if there is an option to make that change to the system? I have been trying but not successful. I found the yourtube.py but cannot find the place to change it.

[meeb]

@Lucas319315 that edit would be made in the settings.py file, not youtube.py. Add postprocessors as a key to YOUTUBE_DEFAULTS here:

https://github.com/meeb/tubesync/blob/main/tubesync/tubesync/settings.py#L157

[Lucas319315]

@Lucas319315 that edit would be made in the settings.py file, not youtube.py. Add postprocessors as a key to YOUTUBE_DEFAULTS here:

https://github.com/meeb/tubesync/blob/main/tubesync/tubesync/settings.py#L157

Thanks for the assist!

[mylogon341]

Is it possible we're able to have an incremental release with this fix in please so we're not all having to manually edit files? That would be great.

[meeb]

There isn't a fix in the repo for this yet @mylogon341 - I've not been able to replicate the issue to debug it. If anyone has a bare SQLite database from a tubesync install which exhibits this behaviour feel free to add it to the issue.

[mylogon341]

Is it possible we're able to have an incremental release with this fix in please so we're not all having to manually edit files? That would be great.

Sorry - ignore. I got myself mixed up on bugs I'm following and just added a comment in my 'rona recovery. In my head this was for a ticket for sponsor block not working properly. Maybe this is related and maybe not 🤷‍♂️ I'll maybe come back to this and do some more testing when my brain-fog clears.

[duloo0]

I'm pretty confused about how to edit that settings.py file. I'd really like to get SponsorBlock to work.

[Lucas319315]

I'm pretty confused about how to edit that settings.py file. I'd really like to get SponsorBlock to work.

What are you using to host docker? I was using unraid and ended up using console and then was able to run commands on it.

Here is what i had to do to get it edited. apt update apt upgrade apt-get install nano cd tubesync nano settings.py

[meeb]

@duloo0 you should just be able to tick the sponsorblock categories from the edit source form, unless you are impacted with this bug.

[duloo0]

I'm impacted by the same bug unfortunately.

[meeb]

How big is your SQLite database? If you can upload it so I can replicate the issue I can fix why the option isn't working.

[thavelick]

Hi, I'm having this issue, or at least a similar one where sponsorblock isn't working. I have Sponsorblock enabled with:

What blocked?: | Sponsor: Intermission/Intro Animation: Endcards/Credits: Unpaid/Self Promotion: Preview/Recap: Filler Tangent: Interaction Reminder: Non-Music Section:

And I get:

2023-12-11 23:23:46,514 [tubesync/INFO] Scheduling task to download thumbnail for: But What IS A Lens Flare? from: https://i.ytimg.com/vi/z_WRaTsGbVE/maxresdefault.jpg
172.17.0.1 - thavelick [11/Dec/2023:16:23:46 -0700] "POST /media-redownload/6394ef85-9cd2-4885-afcc-c41df310c7d1 HTTP/1.0" 302 0 "https://tubesync.tristanhavelick.com/media-redownload/6394ef85-9cd2-4885-afcc-c41df310c7d1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
172.17.0.1 - thavelick [11/Dec/2023:16:23:46 -0700] "GET /media/6394ef85-9cd2-4885-afcc-c41df310c7d1?message=redownloading HTTP/1.0" 200 20270 "https://tubesync.tristanhavelick.com/media-redownload/6394ef85-9cd2-4885-afcc-c41df310c7d1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
172.17.0.1 - thavelick [11/Dec/2023:16:23:46 -0700] "GET /static/images/nothumb.png HTTP/1.0" 304 0 "https://tubesync.tristanhavelick.com/media/6394ef85-9cd2-4885-afcc-c41df310c7d1?message=redownloading" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
2023-12-11 23:23:50,827 [tubesync/INFO] Resizing 1280x720 thumbnail to 430x240: https://i.ytimg.com/vi/z_WRaTsGbVE/maxresdefault.jpg
2023-12-11 23:23:50,869 [tubesync/INFO] Saved thumbnail for: z_WRaTsGbVE from: https://i.ytimg.com/vi/z_WRaTsGbVE/maxresdefault.jpg
2023-12-11 23:23:55,806 [tubesync/INFO] Downloading media: z_WRaTsGbVE (UUID: 6394ef85-9cd2-4885-afcc-c41df310c7d1) to: "/downloads/video/minutephysics/2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.mkv"
2023-12-11 23:23:57,816 [tubesync/INFO] [youtube-dl] downloading: 2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.f248.webm - 55.2% of 35.19MiB at 30.72MiB/s, 00:00 remaining
2023-12-11 23:23:58,352 [tubesync/INFO] [youtube-dl] downloading: 2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.f248.webm - 100.0% of 35.19MiB at 34.36MiB/s, 00:00 remaining
2023-12-11 23:23:58,438 [tubesync/INFO] [youtube-dl] finished downloading: 2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.f248.webm - 35.19MiB in 00:00:01
2023-12-11 23:23:58,913 [tubesync/INFO] [youtube-dl] finished downloading: 2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.f251.webm - 4.90MiB in 00:00:00
ERROR: Object of type CommaSepChoiceField is not JSON serializable
2023-12-11 23:23:59,136 [tubesync/INFO] Successfully downloaded media: z_WRaTsGbVE (UUID: 6394ef85-9cd2-4885-afcc-c41df310c7d1) to: "/downloads/video/minutephysics/2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.mkv"
2023-12-11 23:23:59,172 [tubesync/INFO] Copying media thumbnail from: /config/media/thumbs/63/6394ef85-9cd2-4885-afcc-c41df310c7d1.jpg to: /downloads/video/minutephysics/2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.jpg
2023-12-11 23:23:59,172 [tubesync/INFO] Writing media NFO file to: to: /downloads/video/minutephysics/2023-12-06_minutephysics_but-what-is-a-lens-flare_z_WRaTsGbVE_1080p-vp09-opus.nfo

in my logs when I try to redownload a video. The key thing seems to be the ERROR: Object of type CommaSepChoiceField is not JSON serializable error.

Also happens with just the "sponsor" option checked along with "enable sponsorblock". As far as I can tell, digging around , we just need to change:

 sbopt = {
        'key': 'SponsorBlock',
        'categories': [sponsor_categories]
    }

to

 sbopt = {
        'key': 'SponsorBlock',
        'categories': sponsor_categories.get_my_choices()
    }

But that's just my impression from looking at the code and is entirely untested. I'll try to play with it and possible post a PR sometime over the next week or so.

My DB is like a gig, so it probably isn't feasible to just upload. If I can't confirm or figure out a fix, I'll spin up an instance with a pared down version of the problem and post the db

[thavelick]

@meeb Okay, I was able to reproduce this on main (currently 6853c1fa76ba0c2a26d62477f3e2a5bf2233bb2e) with a small database which I've attached.

tubesync-sponsorblock-broken-db.zip

To reproduce the ERROR: Object of type CommaSepChoiceField is not JSON serializable problem:

1. spin up a container built against the above commit with the attached db
2. Open the website
3. Click Media
4. Click the single video (a ClimateTown video about climate change)
5. Click "Redownload Media"
6. Make sure you're tailing the logs per the README
7. Click the button to confirm
8. You should see something like this in the logs:

tubesync-1  | 2023-12-12 01:50:30,013 [tubesync/INFO] [youtube-dl] finished downloading: 2023-10-30_climatetown_chicago-doesnt-own-its-own-streets-climate-town_fDx6no-7HZE_1080p-vp09-opus.f248.webm - 342.07MiB in 00:01:02
tubesync-1  | 2023-12-12 01:50:37,476 [tubesync/INFO] [youtube-dl] finished downloading: 2023-10-30_climatetown_chicago-doesnt-own-its-own-streets-climate-town_fDx6no-7HZE_1080p-vp09-opus.f251.webm - 29.02MiB in 00:00:07
tubesync-1  | ERROR: Object of type CommaSepChoiceField is not JSON serializable
tubesync-1  | 2023-12-12 01:50:38,657 [tubesync/INFO] Successfully downloaded media: fDx6no-7HZE (UUID: 0b7affc5-dd94-4645-8006-e6adebd06e64) to: "/downloads/video/climatetown/2023-10-30_climatetown_chicago-doesnt-own-its-own-streets-climate-town_fDx6no-7HZE_1080p-vp09-opus.mkv"

9. Watch the video, you'll notice that the sponsor is not blocked

I'll continue looking at this, but I'm less confident that my proposed solution will work.

[meeb]

Thanks @thavelick - I've pushed a very minor tweak that may help with this. It'll be in the next release and :latest shortly.

[thavelick]

@meeb I built a container with the above commit and now I'm getting:

tubesync-1  | WARNING: Unable to communicate with SponsorBlock API: HTTP Error 400: Bad Request. Retrying (1/3)...
tubesync-1  | WARNING: Unable to communicate with SponsorBlock API: HTTP Error 400: Bad Request. Retrying (2/3)...
tubesync-1  | WARNING: Unable to communicate with SponsorBlock API: HTTP Error 400: Bad Request. Retrying (3/3)..

Seems like we're getting closer! But sponsor segments are still in there

[meeb]

Ah, well that looks like TubeSync's end is working, I just need to find why the upstream API is blocking the requests then (or find an alternative endpoint).

Edit: or perhaps the category names are wrong and that's what the API is complaining about? Mostly a note for future me to look into.

[meeb]

Yep it was me being an idiot and wrapping a list in another list as a typo. This should be fixed once the above commit builds into :latest.

[irishj]

Can you please commit the builds into latest so I can pull the latest image, ads in my videos are killing me. Thank you !

[meeb]

@irishj this should be in :latest, try pulling an updated image. If it's still not working check your sources sponsorblock categories.

[irishj]

Thanks @meeb - I'll pull the latest and check it out. Thanks again for this great software.

[uduncanu]

I'm having the same issue here, but I don't think those commits are helping. The problem seems to be that it clears the sponsorblock settings whenever it fetches the metadata.

Versions tested: 0.13.3 and master Steps to reproduce:

1. Start up tubesync in a docker container (completely fresh, it can create a new db file): docker run -p 8080:4848 ghcr.io/meeb/tubesync:latest
2. Go to the web interface (on port 8080 using the above command) and go to add a source
3. Enter source information, preferably one that doesn't exist as that'll keep trying to get the metadata. I've been using https://www.youtube.com/c/testyoutubechannelidthathopefullywontexist532454
4. Make sure you set sponsorblock settings, and then hit save.
5. Wait a few seconds for it to try fetching metadata
6. Go back to the source and the sponsorblock settings are gone.
7. If you re-add them and save, wait a few mins for the next metadata fetch to try and it'll remove them again.

I tried digging into it a bit more and it looked like it was the bit in the index_source_task function in tasks.py that saves the source after retrieving it and setting has_failed to false. That then looked like it was a typing issue, as Django seems to expect CommaSepChoiceField's from_db_value to return a list but it actually returns an object. Changing it to a list fixes the issue with settings changing every time it fetches metadata, but it breaks the user interface and probably loads of other things, so I gave up at that point!

[meeb]

Thanks for investigating! I'll take a look at your reproduction steps.

[irishj]

I'm seeing a similar issue with existing sources which have already been added.

I can set the sponsor block options via the GUI or via a MySQL query (I'm using MySQL as the back-end DB) and the chosen block options are saved, but when the scheduled tasks are run, the sponsor block options are removed for all sources.

The sponsor block enabled option is still enabled, but the sponsor block options for each source have all been removed.

[meeb]

Yeah there's clearly a bug in the handling of the sponsorblock categories. I'll look into it when I get time.