Memory usage during db migration

[eeveelution1313]

I'm in the middle of migrating from sqlite to mariadb, in hopes of speeding some things up, and i'm seeing extremely high memory utilisation. I'm not sure if it could be related to it not being done with scheduled tasks before trying to switch.

screenshot of stats before start of migration

i followed the guide from here https://github.com/meeb/tubesync/blob/main/docs/other-database-backends.md

Not sure if it matters but it's running using podman-compose with the following

  TubeSync:
    image: ghcr.io/meeb/tubesync:latest
    container_name: TubeSync
    ports:
      - 4848:4848
    volumes:
      - ./Managers/TubeSync:/config
      - /mnt/HDD/Media/Youtube/:/downloads
    environment:
      - TZ=Europe/Copenhagen
      - PUID=1000
      - PGID=1907200513
      - TUBESYNC_WORKERS=1
      - DATABASE_CONNECTION=mysql://user:pass@ip:3306/tubesync
    restart: unless-stopped

Very new to github, sorry if i didn't include something needed

[eeveelution1313]

additionally after hitting the max memory it can use it seems to process scheduling tasks in patches slowing down significantly

sorry if it comes off as me complaining, that is not my intention i'm just not sure if it could be valuable information

[meeb]

This is the Django dumpdata / loaddata process. This can use a lot of memory. Your usage does seem extreme however. How big is the JSON file you're trying to import into MySQL?

[eeveelution1313]

i just started using this, and pretty much just added all of my subscriptions in one go, as i'm getting tired of youtube's interface. The json is 1.1 gig, the sqlite is also roughly the same size

[meeb]

Did you solve this? Need any additional advice?

[eeveelution1313]

It did finish, took quite a while, still have times when running where it ends up trying to eat all ram from the system, but overall it works

[meeb]

Excellent thanks. The act of indexing large channels shuffles about massive amounts of data so when they trigger it'll use quite a lot of RAM unfortunately. I'll close this for now but feel free to comment again if you have further issues.