dolphin icon indicating copy to clipboard operation
dolphin copied to clipboard

`Maximum memory usage` is wrong for parallel bursts

Open scottstanie opened this issue 2 years ago • 0 comments

with 3 parallel bursts on a small test set (using the GPU):

dolphin config --slc ../gslcs/t078_16557*/*/t*h5 -s 5 3  --threads 16 --n-work 2 --n-parallel 3

The final print out was

                      INFO     Maximum memory usage: 4.03 GB

Moving down to 1 parallel burst:

dolphin config --slc ../gslcs/t078_16557*/*/t*h5 -s 5 3 --threads 16 --n-work 2 --n-parallel 1

Result:

                      INFO     Maximum memory usage: 7.56 GB

There's a very small chance this isn't wrong... but it doesn't make sense, as I would expect 3x the RAM used for 3 burst processes running concurrently. This might be easier to debug if we had a smaller dedicated machine to watch the actual RAM usage.

scottstanie avatar Sep 07 '23 15:09 scottstanie