cctbx_project
cctbx_project copied to clipboard
Run_tests_parallel uses only one core
On Perlmutter and friends, run_tests_parallel runs tests in parallel, but all tests are run on just one core.
I created a small reproducer that narrows it down to certain module imports.
main script:
import subprocess
from multiprocessing import Pool
commands = [["libtbx.python", "dummy.py"]]*10
pool = Pool(processes=10)
for cmd in commands:
pool.apply_async(subprocess.Popen(cmd))
pool.close()
pool.join()
dummy.py (the import are not important, they just to show that python import run fine)
from boost_adaptbx import boost
#import boost_adaptbx.boost.python as bp
#import boost_python_meta_ext
#import boost_tuple_ext
import inspect
import os
import re
import sys
import warnings
import numpy as np
from libtbx import cpp_function_name
x = 0
for i in range(10**8):
x += 1.3*i
Without the comments, the dummy scripts run on 10 cores. With either one, it's down to one core.
Wrapping the import in os.sched_getaffinity
and os.sched_setaffinity
helps, but is not a real solution.