Expectations on CPU usage and number of processes?

[qacollective]

Hello,

Just writing in to check my expectations on what the timescaledb-parallel-copy script should look like while running.

I am running it inside a docker container but that container doesn't have any restrictions on memory or CPU usage. I have matched the number of workers to the number of CPU cores. I am only however seeing minimal (10-20%) CPU usage, and cannot see more than one process running in the 'top' output.

Is this to be expected?

This is the command I'm running from within a Python script:

/bin/bash -c 'timescaledb-parallel-copy --connection "postgres://postgres:****@localhost:5432/trade" --schema XXX --table YYY --file file.csv --workers 8 --reporting-period 30s --skip-header'

This is typical top output:

top - 11:46:02 up 4 days, 12:10,  0 users,  load average: 4.13, 4.58, 4.86
Tasks:   5 total,   1 running,   4 sleeping,   0 stopped,   0 zombie
%Cpu(s): 22.8 us, 12.3 sy,  0.2 ni, 59.2 id,  4.5 wa,  0.0 hi,  1.0 si,  0.0 st
KiB Mem : 32824804 total,   946600 free, 12929264 used, 18948940 buff/cache
KiB Swap:  2097148 total,        0 free,  2097148 used. 10609320 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                     
      7 winer     20   0 1373344  27052   6212 S  10.6  0.1   0:04.85 timescaledb-par                                                                                                                             
      1 winer     20   0   96640  20192  12084 S   0.0  0.1   0:00.09 python3                                                                                                                                     
      6 winer     20   0    4624    784    716 S   0.0  0.0   0:00.00 sh                                                                                                                                          
     23 winer     20   0   18504   3344   2968 S   0.0  0.0   0:00.02 bash                                                                                                                                        
     37 winer     20   0   38724   3156   2664 R   0.0  0.0   0:00.00 top           

This is some sample program output:

2020-10-04 11:52:25,278 - TickDataLoader - INFO - at 30s, row rate 79833.11/sec (period), row rate 79833.11/sec (overall), 2.395000E+06 total rows
2020-10-04 11:52:55,278 - TickDataLoader - INFO - at 1m0s, row rate 76833.29/sec (period), row rate 78333.20/sec (overall), 4.700000E+06 total rows
2020-10-04 11:53:25,278 - TickDataLoader - INFO - at 1m30s, row rate 85500.03/sec (period), row rate 80722.14/sec (overall), 7.265000E+06 total rows
2020-10-04 11:53:55,278 - TickDataLoader - INFO - at 2m0s, row rate 93999.98/sec (period), row rate 84041.60/sec (overall), 1.008500E+07 total rows
2020-10-04 11:54:25,278 - TickDataLoader - INFO - at 2m30s, row rate 94333.36/sec (period), row rate 86099.95/sec (overall), 1.291500E+07 total rows
2020-10-04 11:54:55,279 - TickDataLoader - INFO - at 3m0s, row rate 101999.91/sec (period), row rate 88749.94/sec (overall), 1.597500E+07 total rows
2020-10-04 11:55:25,286 - TickDataLoader - INFO - at 3m30s, row rate 104166.73/sec (period), row rate 90952.34/sec (overall), 1.910000E+07 total rows
2020-10-04 11:55:55,278 - TickDataLoader - INFO - at 4m0s, row rate 99333.38/sec (period), row rate 91999.97/sec (overall), 2.208000E+07 total rows
2020-10-04 11:56:25,278 - TickDataLoader - INFO - at 4m30s, row rate 99833.33/sec (period), row rate 92870.34/sec (overall), 2.507500E+07 total rows
2020-10-04 11:56:55,278 - TickDataLoader - INFO - at 5m0s, row rate 93333.32/sec (period), row rate 92916.64/sec (overall), 2.787500E+07 total rows
2020-10-04 11:57:25,278 - TickDataLoader - INFO - at 5m30s, row rate 102166.62/sec (period), row rate 93757.55/sec (overall), 3.094000E+07 total rows
2020-10-04 11:57:55,278 - TickDataLoader - INFO - at 6m0s, row rate 102666.66/sec (period), row rate 94499.97/sec (overall), 3.402000E+07 total rows
2020-10-04 11:58:25,278 - TickDataLoader - INFO - at 6m30s, row rate 106166.71/sec (period), row rate 95397.41/sec (overall), 3.720500E+07 total rows
2020-10-04 11:58:55,278 - TickDataLoader - INFO - at 7m0s, row rate 107833.29/sec (period), row rate 96285.69/sec (overall), 4.044000E+07 total rows
2020-10-04 11:59:25,278 - TickDataLoader - INFO - at 7m30s, row rate 105666.75/sec (period), row rate 96911.09/sec (overall), 4.361000E+07 total rows
2020-10-04 11:59:55,278 - TickDataLoader - INFO - at 8m0s, row rate 104499.99/sec (period), row rate 97385.40/sec (overall), 4.674500E+07 total rows
2020-10-04 12:00:25,278 - TickDataLoader - INFO - at 8m30s, row rate 105499.99/sec (period), row rate 97862.73/sec (overall), 4.991000E+07 total rows
2020-10-04 12:00:55,278 - TickDataLoader - INFO - at 9m0s, row rate 103500.00/sec (period), row rate 98175.91/sec (overall), 5.301500E+07 total rows
2020-10-04 12:01:25,278 - TickDataLoader - INFO - at 9m30s, row rate 102666.50/sec (period), row rate 98412.26/sec (overall), 5.609500E+07 total rows
2020-10-04 12:01:55,278 - TickDataLoader - INFO - at 10m0s, row rate 107500.01/sec (period), row rate 98866.65/sec (overall), 5.932000E+07 total rows
2020-10-04 12:02:25,278 - TickDataLoader - INFO - at 10m30s, row rate 106500.19/sec (period), row rate 99230.15/sec (overall), 6.251500E+07 total rows
2020-10-04 12:02:55,278 - TickDataLoader - INFO - at 11m0s, row rate 108333.32/sec (period), row rate 99643.93/sec (overall), 6.576500E+07 total rows
2020-10-04 12:03:25,278 - TickDataLoader - INFO - at 11m30s, row rate 106666.69/sec (period), row rate 99949.26/sec (overall), 6.896500E+07 total rows
2020-10-04 12:03:55,278 - TickDataLoader - INFO - at 12m0s, row rate 108166.62/sec (period), row rate 100291.65/sec (overall), 7.221000E+07 total rows
2020-10-04 12:04:25,278 - TickDataLoader - INFO - at 12m30s, row rate 106666.62/sec (period), row rate 100546.65/sec (overall), 7.541000E+07 total rows
2020-10-04 12:04:55,278 - TickDataLoader - INFO - at 13m0s, row rate 108666.68/sec (period), row rate 100858.96/sec (overall), 7.867000E+07 total rows
2020-10-04 12:05:25,278 - TickDataLoader - INFO - at 13m30s, row rate 111000.01/sec (period), row rate 101234.56/sec (overall), 8.200000E+07 total rows
2020-10-04 12:05:55,279 - TickDataLoader - INFO - at 14m0s, row rate 106994.01/sec (period), row rate 101440.26/sec (overall), 8.521000E+07 total rows
2020-10-04 12:06:25,278 - TickDataLoader - INFO - at 14m30s, row rate 109339.51/sec (period), row rate 101712.63/sec (overall), 8.849000E+07 total rows
2020-10-04 12:06:55,278 - TickDataLoader - INFO - at 15m0s, row rate 114666.39/sec (period), row rate 102144.43/sec (overall), 9.193000E+07 total rows
2020-10-04 12:07:25,278 - TickDataLoader - INFO - at 15m30s, row rate 112500.16/sec (period), row rate 102478.48/sec (overall), 9.530500E+07 total rows

[yazeed]

Looking for the same information here as well.