Performance issues on dump

[cterence]

Hello,

I've made some basic testing of Replibyte and I'm currently experiencing two performance issues :

• Dumping operations are taking way longer than expected, around 0.5MB/s
• The RAM consumption gets out of control after a while, progressively increasing until generating an OOMKill

Here are the testing conditions :

Hardware :

• Lenovo Thinkpad T14s Gen 1
  • RAM : 16GB
  • CPU : AMD Ryzen 7 PRO 4750U w/ 8 physical core, 16 virtual cores
  • NVMe SSD : (300MB/s write, 700MB/s read)
  • OS : Linux, PopOS 20.04

Software :

• Replibyte version : 0.10.0
• Replibyte config replibyte.yaml

  source:
    connection_uri: postgres://postgres:postgres@localhost:5433/demo
  datastore:
    local_disk:
      dir: ./dumps
  destination:
    connection_uri: postgres://postgres:postgres@localhost:5432/demo
  

• pg_dump version : pg_dump (PostgreSQL) 15.2 (Ubuntu 15.2-1.pgdg20.04+1)
• docker-compose.yaml

  services:
    postgresql-prod:
      container_name: postgresql-prod
      image: postgres:11
      environment:
        - POSTGRES_USER=postgres
        - POSTGRES_PASSWORD=postgres
      ports:
        - 5433:5432
      volumes:
        - ./sample-data:/docker-entrypoint-initdb.d
  
    postgresql-staging:
      container_name: postgresql-staging
      image: postgres:11
      ports:
        - 5432:5432
      environment:
        - POSTGRES_USER=postgres
        - POSTGRES_PASSWORD=postgres
  

• Sample PostgreSQL data : https://edu.postgrespro.com/demo-small-en.zip (around 320MB of data when loaded in Postgres)
• Commands run : replibyte --config ./replibyte.yaml dump create & replibyte --config ./replibyte.yaml dump restore remote --value latest

With all of this, here are my testing steps :

• Deploy the stack & load the sample data
• Creating the dump with replibyte

Creating the dump takes around 11 min at 0.4~0.5 MB/s.

Is there anything that I'm doing wrong ?

Don't hesitate to ask for more details if I've gone over anything, I'll be happy to help.

[TheKipmaster]

I've been having the same issue in a very similar environment. Any word on this?

[maxleroy]

Yup pretty much the same happening here, maybe worse. Running on a ubuntu cloud vm 2vCPUs/4GB RAM, with a local 200MB MySQL database. I get OOM killed with replibyte using 3GB of memory 😬

[ikegentz]

Slow for me as well, even on local disk I'm getting < 1Mib/sec when creating a dump

[evoxmusic]

Hello 👋, I'm aware of this issue and I'm working on benchmarking and performance improvements here. Thanks for your feedback