Atkinson-Lab
Unexpectedly high thread count / CPU usage
Hello,
We are seeing each parallel worker for Tractor try to use all 64 threads available on the machine:
This feels extremely similar to other R applications we've seen managed with Conda environments, where the default behavior of BLAS implementation you've selected (OpenBLAS) tries to use all of the cores on the machine even when this is very inefficient: https://github.com/weizhouUMICH/SAIGE/issues/380
Is the usage of many cores for each worker intentional or accidental? If accidental, we can fix it using https://cran.r-project.org/web/packages/RhpcBLASctl/index.html or other tools. If intentional, how many threads should we allocate for each worker? 8? 16?
While I'm here: We also noticed after time that each of these worker processes topped out at 32GB of RAM used exactly.
Is this RAM usage configurable? The fact that each one used exactly 32GB despite input file size feels like it must be a manually controlled setting somewhere.