cctbx_project icon indicating copy to clipboard operation
cctbx_project copied to clipboard

Run_tests_parallel uses only one core

Open Trzs opened this issue 5 months ago • 5 comments

On Perlmutter and friends, run_tests_parallel runs tests in parallel, but all tests are run on just one core.

I created a small reproducer that narrows it down to certain module imports.

main script:

import subprocess
from multiprocessing import Pool

commands = [["libtbx.python", "dummy.py"]]*10

pool = Pool(processes=10)
for cmd in commands:
    pool.apply_async(subprocess.Popen(cmd))
pool.close()
pool.join()

dummy.py (the import are not important, they just to show that python import run fine)

from boost_adaptbx import boost
#import boost_adaptbx.boost.python as bp
#import boost_python_meta_ext
#import boost_tuple_ext

import inspect
import os
import re
import sys
import warnings
import numpy as np

from libtbx import cpp_function_name

x = 0
for i in range(10**8):
    x += 1.3*i

Without the comments, the dummy scripts run on 10 cores. With either one, it's down to one core. Wrapping the import in os.sched_getaffinity and os.sched_setaffinity helps, but is not a real solution.

Trzs avatar Jan 12 '24 18:01 Trzs