pgcopydb
pgcopydb copied to clipboard
dimitri
Processes become frozen
In addition to #695 I rebult the program from the main branch code
The processes have been working for more than 7 hours. It seems like freeze.
I had to kill processes. The log after killing is here pgcopydb_05_03_2024_23_50_00.log
Last record in the log before killing was 2024-03-05 23:50:32.870 14192 INFO sequences.c:245 Reset sequences values on the target database
The sizes of DBs are almost the same. But target db looks corrupted,
target db szie 12828812
source db size 12891064
Hi @FoxMoorhuhn ; could you use gdb and get a stacktrace of some of these pids that look frozen? It might just be that they have a lot of work to do and it takes time given the resources allocated to the pgcopydb instance? can you have a look at top or htop or dstat to see if the process are active during that period of time?
The logs you're sharing show nothing interesting. Consider using --debug for investigation purposes.
Lastly, I find “corrupted” a joker word that ended-up meaning nothing serious at all. Well at least to understand it we need to be able to compare what is expected and what is observed. Can you please give more details about the “corruption” you're mentioning here?
Hi!
"corrupted" is means that although the DB has opened but I am not sure about the integrity of it, because my app works with errors.
I had to kill processes again after 15 hours. The log is here pgcopydb_06_03_2024_17_38_38.zip
htop
ps
Hi, @dimitri
Unfortunately, I don't know how to use gdb to get a stacktrace. I'm not developer, I'm DBA. Perhaps I can do it if you help me. I think it will be interesting for both of us.
I did a series of experiments. and I found a dependence between the size of the DB (count of files) and the count of "jobs" (--table and --index). The results are shown in the Excel-table I have two source DBs: 170 Gb (15K files) and 13 Gb (45K files) My command line is pgcopydb clone --table-jobs=X --index-jobs=Y --source="..." --target="...." 2>pgcopydb$(date+ "%d_%m_%Y_%H_%M_%S").log_
I also attach all the logs pgcopydb.xlsx logs.zip
