Stuck on "Updating metadata from indexing results" – tasks pile up, no further processing

[tehKNi]

Describe the bug Since about a week ago (mid-June 2025), TubeSync started exhibiting an issue where indexing appears to complete normally, but then gets stuck on Updating metadata from indexing results. At this point, the relevant tasks never run, and new "updating metadata" tasks either pile up or block all further operations. Only a reset-tasks seems to temporarily restore activity, but the issue quickly returns.

Environment

• TubeSync: 0.15.6
• yt-dlp: 2025.06.09
• FFmpeg: N-119898-ge6fb8f373e-20250616
• PostgreSQL 15.2 (Docker)

Docker Compose excerpt

tubesync:
  image: ghcr.io/meeb/tubesync:latest
  environment:
    - TUBESYNC_WORKERS=2
    - TUBESYNC_DEBUG=True
    - DATABASE_CONNECTION=postgresql://postgres:<password>@tubesync-db:5432/tubesync
  depends_on:
    - tubesync-db

tubesync-db:
  image: postgres:15.2
  environment:
    - POSTGRES_USER=postgres
    - POSTGRES_PASSWORD=<password>

To Reproduce

1. Start indexing a source with many videos (playlist/channel)
2. Observe logs: items are fetched and indexed
3. See Updating metadata from indexing results for: "<video_id>": <video_title>
4. Task says "Task will run immediately", but it never runs
5. All subsequent tasks remain stuck in this state
6. Only a reset-tasks temporarily helps, but issue recurs immediately

Logs

[INFO] Found 121 media items for source: <source>
[DEBUG] Indexed media: 1: <source> / <video_id>
...
[DEBUG] Indexed media: 121: <source> / <video_id>
[INFO] Updating metadata from indexing results for: "<video_id>": <video_title>
Task will run immediately

No further log entries, and TubeSync appears "stuck" until forced reset even with debug logs set to True.

What I tried

• Restarted containers
• Ran reset-tasks (only temporary effect)
• Enabled debug mode (no new clues, logs simply stop)
• Observed: direct Downloading metadata for: <video_id> tasks do work, but all "updating metadata" tasks never run

Additional notes

• No database, config, or disk errors are reported
• Problem started recently after like a year of stable use
• Only affects "updating metadata" tasks afaik

Request Any idea how to debug this further or what might have changed recently? Is there a way to clear out "stuck" updating tasks, or is something blocking the queue/thread?

[tcely]

what might have changed recently?

What changed recently is the addition of a new system^django-huey for running tasks.

Please be sure to pull the latest image again.

Updating metadata^task should be happening in a new queue using huey.

Any idea how to debug this further

[!IMPORTANT] Please show us any changes that you made to settings.py or local_settings.py, then check your /config/tasks directory where SQLite databases holding task results are stored.

[!TIP] Try removing TUBESYNC_WORKERS as each queue is now configured according to its purpose^queues and tasks from different queues might run independently anyway.

For debugging, there is a useful bit of code from #1109:

$ python3 /app/manage.py shell

import sync.tasks
from pprint import pp
from django_huey import DJANGO_HUEY
from common.huey import h_q_tuple
h_q_names = DJANGO_HUEY.get('queues', dict()).keys()
pp( [ h_q_tuple(q) for q in reversed(h_q_names) ] )

The information important for that task looks like this:

 ('huey_database',
  ['sync.tasks.schedule_indexing', 'sync.tasks.migrate_to_metadata'],
  {'scheduled': (0, []),
   'pending': (0, []),
   'running': set(),
   'results': (0, [])})

Task says "Task will run immediately", but it never runs

[!NOTE] By the way, all that "immediately" means on the tasks page is that the scheduled time is before now. It should not be interpreted as any guarantee about when things are actually executed.

PostgreSQL 15.2 (Docker)

[!TIP] That version is very out of date. More information is available at:

• https://github.com/meeb/tubesync/blob/main/docs/other-database-backends.md
• #961

[tehKNi]

Thanks for the quick feedback!

I removed the TUBESYNC_WORKERS environment variable from my compose (as you suggested) and this seems to have completely resolved the "stuck tasks" issue. Since making this change, TubeSync has been running smoothly for almost two days, with no more metadata update blocks or hanging queues.

I also took the opportunity to upgrade my PostgreSQL container to a newer version, following your documentation.

Thanks again for the help and detailed pointers!

[tcely]

I've added a logged warning about TUBESYNC_WORKERS in #1136.

It should really be removed from everywhere in the repository. Using that setting hasn't been helpful since before #924 was closed.

I've opened #1144 to remove that variable from the README.

Also, #1145 to stop it from having any effects.

You can read more about why using that old variable was a bad idea at:

• https://github.com/meeb/tubesync/issues/927#issuecomment-2784000291

Please close this issue when you're done.