meeb
New source - no crawl
Yesterday I created a new source to troubleshoot some videos that were not downloading from the existing source. I restarted the container and cleared the tasks manually (docker exec -ti tubesync python3 /app/manage.py reset-tasks) but there are no tasks attached to this source. I added debug information to the container but there is no information in the logs.
Index schedule | Every hour Download media? checkmark Created | 2022-04-03 21:14:09 Last crawl | Never
How do I troubleshoot this?
You could try searching for your source UUID in the container logs. That should give you an indication as to what and what is not scheduled at least. What's in your scheduled tasks list under the tasks page?
Under 'Completed tasks' for this source.
There have been no completed tasks that match the specified source filter.
There is no entry for this source under scheduled tasks.
From the logs (using UUID - 1a743f93 - trimmed the value): I removed fonts and styles GET calls.
460:M 04 Apr 2022 07:18:49.508 * Ready to accept connections [2022-04-04 07:18:49 -0500] [464] [INFO] Starting gunicorn 20.1.0 [2022-04-04 07:18:49 -0500] [464] [INFO] Listening at: http://127.0.0.1:8080 (464) [2022-04-04 07:18:49 -0500] [464] [INFO] Using worker: sync [2022-04-04 07:18:49 -0500] [490] [INFO] Booting worker with pid: 490 [2022-04-04 07:18:49 -0500] [491] [INFO] Booting worker with pid: 491 [2022-04-04 07:18:49 -0500] [492] [INFO] Booting worker with pid: 492
37: 192.168.86.42 - - [04/Apr/2022:07:18:52 -0500] "GET /sources HTTP/1.1" 200 2337 "http://192.168.86.226:4848/source/1a743f93-xxxx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
43: 192.168.86.42 - - [04/Apr/2022:07:19:02 -0500] "GET /source/1a743f93-xxxx HTTP/1.1" 200 3170 "http://192.168.86.226:4848/sources" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
49: 192.168.86.42 - - [04/Apr/2022:07:19:13 -0500] "GET /source-update/1a743f93-xxxx HTTP/1.1" 200 4340 "http://192.168.86.226:4848/source/1a743f93-xxxx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
55: 192.168.86.42 - - [04/Apr/2022:07:19:39 -0500] "POST /source-update/1a743f93-xxxx HTTP/1.1" 302 0 "http://192.168.86.226:4848/source-update/1a743f93-xxxx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36" 56: 192.168.86.42 - - [04/Apr/2022:07:19:39 -0500] "GET /source/1a743f93-xxxx?message=source-updated HTTP/1.1" 200 3219 "http://192.168.86.226:4848/source-update/1a743f93-xxxx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
62: 192.168.86.42 - - [04/Apr/2022:07:19:45 -0500] "GET /tasks-completed?filter=1a743f93-xxxx HTTP/1.1" 200 1985 "http://192.168.86.226:4848/source/1a743f93-xxxx?message=source-updated" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
68: 192.168.86.42 - - [04/Apr/2022:07:19:53 -0500] "GET /sources HTTP/1.1" 200 2337 "http://192.168.86.226:4848/tasks-completed?filter=1a743f93-xxxx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
80: 192.168.86.42 - - [04/Apr/2022:07:43:13 -0500] "GET /source/1a743f93-xxxx HTTP/1.1" 200 3165 "http://192.168.86.226:4848/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36" 81: 192.168.86.42 - - [04/Apr/2022:07:43:13 -0500] "GET /source/1a743f93-xxxx HTTP/1.1" 200 3165 "http://192.168.86.226:4848/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
I tried resetting tasks and this time following error was thrown. Btw, the new source name is not shown here.
root@Tower:/mnt/user/downloads# docker exec -ti tubesync python3 /app/manage.py reset-tasks 2022-04-04 11:33:40,772 [tubesync/INFO] Resettings all tasks... 2022-04-04 11:33:41,378 [tubesync/INFO] Resetting tasks for source: nw 2022-04-04 11:34:10,313 [tubesync/INFO] Resetting tasks for source: Favs 2022-04-04 11:34:33,245 [tubesync/INFO] Resetting tasks for source: nw_fa Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.9/dist-packages/django/db/backends/sqlite3/base.py", line 423, in execute return Database.Cursor.execute(self, query, params) sqlite3.OperationalError: database is locked
Traceback (most recent call last):
File "/app/manage.py", line 18, in
Ah, it's probably the lock issue then which is why the original crawl task didn't get scheduled. Tasks are stored in the database so if that failed to write it wouldn't be scheduled. This is a side-effect of using SQLite as a default for the database with a front end and a background worker which can both write to the database. When you get enough media and sources you end up having write lock issues with SQLite. If you have quite a bit of media and quite a lot of sources I would suggest you try migrating to using Postgres as a backend. Most of your scheduling and task issues with generally be mitigated post-migration.
Assuming you've resolved this by migrating to a Postgres or MySQL backend, I'll close this for now. Feel free to re-open the issue if you still run into this.