bitmagnet Gorm and queue errors

[x] I have checked the existing issues to avoid duplicates
[x] I have redacted any info hashes and content metadata from any logs or screenshots attached to this issue

Describe the bug

After 1,010,731 new torrent distributions are not added

To Reproduce

Steps to reproduce the behavior:

docker compose up -d

Details


services:
  bitmagnet:
    image: ghcr.io/bitmagnet-io/bitmagnet:latest
    container_name: bitmagnet
    ports:
      # API and WebUI port:
      - "3333:3333"
      # BitTorrent ports:
      - "3334:3334/tcp"
      - "3334:3334/udp"
    restart: unless-stopped
    environment:
      - POSTGRES_HOST=postgres
      - POSTGRES_PASSWORD=123
    command:
      - worker
      - run
      - --keys=http_server
      - --keys=queue_server
      # disable the next line to run without DHT crawler
      - --keys=dht_crawler
    depends_on:
      postgres:
        condition: service_healthy

  postgres:
    image: postgres:16-alpine
    container_name: bitmagnet-postgres
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    ports:
      - "55323:5432"
    restart: unless-stopped
    environment:
      - POSTGRES_PASSWORD=123
      - POSTGRES_DB=bitmagnet
      - PGUSER=postgres
    shm_size: 1g
    healthcheck:
      test:
        - CMD-SHELL
        - pg_isready
      start_period: 20s
      interval: 10s

Expected behavior

A clear and concise description of what you expected to happen.

Environment Information (Required)

Bitmagnet version: v0.9.3
OS and version: ubuntu docker
Browser and version (if issue is with WebUI): [e.g. Chrome 117.0.5938.149]
Please specify any config values for which you have overridden the defaults: (e.g. dht_crawler.save_pieces: true)

Additional context

logs

bitmagnet    | ERROR    queue   server/server.go:230    error handling joberrorERROR: duplicate key value violates unique constraint "queue_jobs_fingerprint_status_idx" (SQLSTATE 23505)    {"queue": "process_torrent"}
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob
bitmagnet    |  /src/internal/queue/server/server.go:230
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2
bitmagnet    |  /src/internal/queue/server/server.go:169
bitmagnet    | ERROR    queue   server/server.go:214    job failed      {"queue": "process_torrent", "error": "job failed to process: runtime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused"}
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.func1
bitmagnet    |  /src/internal/queue/server/server.go:214
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.(*Query).Transaction.func2
bitmagnet    |  /src/internal/database/dao/gen.go:196
bitmagnet    | gorm.io/gorm.(*DB).Transaction
bitmagnet    |  /go/pkg/mod/gorm.io/[email protected]/finisher_api.go:651
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction
bitmagnet    |  /src/internal/database/dao/gen.go:196
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob
bitmagnet    |  /src/internal/queue/server/server.go:180
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2
bitmagnet    |  /src/internal/queue/server/server.go:169
bitmagnet    | ERROR    gorm    logger/logger.go:72     gorm trace      {"location": "/src/internal/queue/server/server.go:226", "error": "ERROR: duplicate key value violates unique constraint \"queue_jobs_fingerprint_status_idx\" (SQLSTATE 23505)", "elapsed": 17.81044, "sql": "UPDATE \"queue_jobs\" SET \"status\"='retry',\"run_after\"='2024-06-22 18:50:06.889',\"ran_at\"='2024-06-22 18:49:50.888',\"error\"='job failed to process: runtime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused' WHERE \"id\" = '79d22423-65cd-****'", "rows": 0}
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/database/logger.(*customLogger).Trace
bitmagnet    |  /src/internal/database/logger/logger.go:72
bitmagnet    | gorm.io/gorm.(*processor).Execute
bitmagnet    |  /go/pkg/mod/gorm.io/[email protected]/callbacks.go:134
bitmagnet    | gorm.io/gorm.(*DB).Updates
bitmagnet    |  /go/pkg/mod/gorm.io/[email protected]/finisher_api.go:411
bitmagnet    | gorm.io/gen.(*DO).Updates
bitmagnet    |  /go/pkg/mod/gorm.io/[email protected]/do.go:728
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.func1
bitmagnet    |  /src/internal/queue/server/server.go:226
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.(*Query).Transaction.func2
bitmagnet    |  /src/internal/database/dao/gen.go:196
bitmagnet    | gorm.io/gorm.(*DB).Transaction
bitmagnet    |  /go/pkg/mod/gorm.io/[email protected]/finisher_api.go:651
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction
bitmagnet    |  /src/internal/database/dao/gen.go:196
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob
bitmagnet    |  /src/internal/queue/server/server.go:180
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2
bitmagnet    |  /src/internal/queue/server/server.go:169
bitmagnet    | ERROR    queue   server/server.go:230    error handling joberrorERROR: duplicate key value violates unique constraint "queue_jobs_fingerprint_status_idx" (SQLSTATE 23505)    {"queue": "process_torrent"}
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob
bitmagnet    |  /src/internal/queue/server/server.go:230
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2
bitmagnet    |  /src/internal/queue/server/server.go:169
bitmagnet    | ERROR    queue   server/server.go:214    job failed      {"queue": "process_torrent", "error": "job failed to process: runtime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused\nruntime error at Path workflows.default.[5].if_else.if_action.find_match: Get \"https://api.themoviedb.org/3/authentication?api_key=9c6689fa83*****\": dial tcp [::1]:443: connect: connection refused"}
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.func1
bitmagnet    |  /src/internal/queue/server/server.go:214
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.(*Query).Transaction.func2
bitmagnet    |  /src/internal/database/dao/gen.go:196
bitmagnet    | gorm.io/gorm.(*DB).Transaction
bitmagnet    |  /go/pkg/mod/gorm.io/[email protected]/finisher_api.go:651
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction
bitmagnet    |  /src/internal/database/dao/gen.go:196
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob
bitmagnet    |  /src/internal/queue/server/server.go:180
bitmagnet    | github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2
bitmagnet    |  /src/internal/queue/server/server.go:169

Jun 22 '24 19:06 Alexandrsv

in my case this is the trace. it isn't related to the number of torrent, I have 7 mil.

log

ERROR	queue	server/server.go:214	job failed	{"queue": "process_torrent", "error": "job failed to process: missing 1 info hashes"}
  github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.func1
    /src/internal/queue/server/server.go:214
  github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.(*Query).Transaction.func2
    /src/internal/database/dao/gen.go:196
  gorm.io/gorm.(*DB).Transaction
    /go/pkg/mod/gorm.io/[email protected]/finisher_api.go:651
  github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction
    /src/internal/database/dao/gen.go:196
  github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob
    /src/internal/queue/server/server.go:180
  github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2
    /src/internal/queue/server/server.go:169
ERROR	gorm	logger/logger.go:72	gorm trace	{"location": "/src/internal/queue/server/server.go:226", "error": "ERROR: duplicate key value violates unique constraint \"queue_jobs_fingerprint_status_idx\" (SQLSTATE 23505)", "elapsed": 9.325193, "sql": "UPDATE \"queue_jobs\" SET \"status\"='retry',\"run_after\"='2024-07-17 13:41:41.372',\"ran_at\"='2024-07-17 13:41:25.372',\"error\"='job failed to process: missing 1 info hashes' WHERE \"id\" = 'faa8de3d-7920-4760-8f58-988d3383cd01'", "rows": 0}
  github.com/bitmagnet-io/bitmagnet/internal/database/logger.(*customLogger).Trace
    /src/internal/database/logger/logger.go:72
  gorm.io/gorm.(*processor).Execute
    /go/pkg/mod/gorm.io/[email protected]/callbacks.go:134
  gorm.io/gorm.(*DB).Updates
    /go/pkg/mod/gorm.io/[email protected]/finisher_api.go:411
  gorm.io/gen.(*DO).Updates
    /go/pkg/mod/gorm.io/[email protected]/do.go:728
  github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.func1
    /src/internal/queue/server/server.go:226
  github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob.(*Query).Transaction.func2
    /src/internal/database/dao/gen.go:196
  gorm.io/gorm.(*DB).Transaction
    /go/pkg/mod/gorm.io/[email protected]/finisher_api.go:651
  github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction
    /src/internal/database/dao/gen.go:196
  github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob
    /src/internal/queue/server/server.go:180
  github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2
    /src/internal/queue/server/server.go:169
ERROR	queue	server/server.go:230	error handling joberrorERROR: duplicate key value violates unique constraint "queue_jobs_fingerprint_status_idx" (SQLSTATE 23505)	{"queue": "process_torrent"}
  github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).handleJob
    /src/internal/queue/server/server.go:230
  github.com/bitmagnet-io/bitmagnet/internal/queue/server.(*serverHandler).start.func2
    /src/internal/queue/server/server.go:169

this part of the db schema is causing the integrity error https://github.com/bitmagnet-io/bitmagnet/blob/6cf825b907f0905100f5f9c5fd18943b26c0833b/migrations/00012_queue.sql#L32

my workaround was to delete the jobs in retry status

delete from public.queue_jobs
where status = 'retry'

Jul 17 '24 13:07 scratchmex

this worked a treat! thanks

Sep 04 '24 09:09 blackcaldera

in my case this is the trace. it isn't related to the number of torrent, I have 7 mil.

log this part of the db schema is causing the integrity error

https://github.com/bitmagnet-io/bitmagnet/blob/6cf825b907f0905100f5f9c5fd18943b26c0833b/migrations/00012_queue.sql#L32

my workaround was to delete the jobs in retry status
delete from public.queue_jobs
where status = 'retry'

Could you let me know where you added this line of code? I'm also getting the same error. Do you put it in the compose file?

Oct 06 '24 16:10 Wofflo

Install something like pgAdmin and run it as a script on the database

Oct 10 '24 06:10 blackcaldera

Install something like pgAdmin and run it as a script on the database

Thank you! I shall try this tonight after work.

Oct 10 '24 06:10 Wofflo

Please note this should not be logged as an error, the error log can be ignored and will be silenced in due course. No querying of the database is needed to remedy this - best to just ignore the error until it is silenced in a future version.

Oct 10 '24 12:10 mgdigital

I eventually managed to get pgAdmin installed on docker and configured (took some time) and managed to run the script. Now after restarting the bitmagnet container stack it's all working again and loading new results :-D thanks for the help everyone!

Oct 10 '24 21:10 Wofflo

Hey, okay so to put it a different way - any issue you guys are having is unrelated to the Gorm duplicate key error. If you're having to clear out the queue then yes it will allow new items to appear but at the expense of old ones not being indexed properly. Could I check - out of people having this issue, does anyone have TMDB enabled with a personal API key? If you haven't configured anything for TMDB you will be using the default rate limited API key and the queue will run a lot slower. If you set a personal API key (or disable TMDB) you should find the queue will quickly work back down to zero and you'll start seeing new items. The rate limiting "shouldn''t" cause the queue to back up like this, it seems to work okay for most users, I'm just wary of opening the flood gates and getting the default API key banned which would break it for everyone who hasn't got a personal key. An alternative would be to disable TMDB integration altogether for users without a personal API key...

Oct 11 '24 06:10 mgdigital

I have a personal api key in use generated from my profile page on TMDB. Is this the right place to get one or do you need to be on the developer side of TMDB to obtain it?

Oct 11 '24 06:10 Wofflo

I have a personal api key in use generated from my profile page on TMDB. Is this the right place to get one or do you need to be on the developer side of TMDB to obtain it?

You can get a key by following the guide here: https://kb.synology.com/en-au/DSM/tutorial/How_to_apply_for_a_personal_API_key_to_get_video_info

You'll know it's been set properly if you don't see this warning on startup:

you are using the default TMDB api key; TMDB requests will be limited to 1 per second; to remove this warning please configure a personal TMDB api key

You can also run bitmagnet config show to check it's been set.

The other possibility is that TMDB is blocked in your country (I suspect from @Alexandrsv 's username they might be in Russia where it is blocked). If this is the case you'd need to disable TMDB altogether with TMDB_ENABLED=false.

It would be helpful to know what the error is on the retry jobs - you can find it in the error column of the queue_jobs table (but you already cleared everything out) - if this happens again please let me know what the error was.

Oct 11 '24 07:10 mgdigital

The duplicate key error should be fixed in https://github.com/bitmagnet-io/bitmagnet/pull/319 I think your queue was running slow due to an issue with TMDB - try either setting a personal API key or disabling TMDB

Oct 15 '24 07:10 mgdigital

I know this issue is closed, but in 0.9.5 I had the same error logs as the OP, torrent count froze at 3M (in another installation it froze at 7M). Checking the metrics, status="pending" was growing infinitely and status="processed" was frozen like status="retry". After executing the mentioned query to clean up status="retry", boom! all the pending torrents were processed until reaching 0 pending. The web UI was also updated with the new torrents.

Feb 25 '25 12:02 plexxman