cppimport icon indicating copy to clipboard operation
cppimport copied to clipboard

Add configuration for setting the degree of compilation parallelism.

Open yaoyuannnn opened this issue 4 years ago • 2 comments

Maybe the first question is how many cppimport will use by default. Since I'm running it on a machine with a small memory, sometimes I can see the compilation goes out of memory.

yaoyuannnn avatar Dec 17 '20 22:12 yaoyuannnn

Currently, parallelism is off by default. If you turn parallelism on, the number of cores is equal to multiprocessing.cpu_count().

This would be quite easy to modify. I have two proposals for how we could do this:

  • Change the parallelism flag to instead take a parameter indicating the number of threads to use. Ideally, this would be done in a way that maintains backward compatibility with the current True/False behavior. So, perhaps, True == multiprocessing.cpu_count, False = 1, and then any integer uses that number of threads.
  • A slightly different, but similar proposal would be to just add another configuration field that contains the number of threads to use.

Some locations in the code that would be useful for this: https://github.com/tbenthompson/cppimport/blob/00c40d9a7f706409156a42ff3a7934e54a8050c1/cppimport/templating.py#L46 https://github.com/tbenthompson/cppimport/blob/00c40d9a7f706409156a42ff3a7934e54a8050c1/cppimport/build_module.py#L57 https://github.com/tbenthompson/cppimport/blob/00c40d9a7f706409156a42ff3a7934e54a8050c1/cppimport/build_module.py#L150

I would be excited to accept a pull request if you have the time to implement this!

tbenthompson avatar Dec 18 '20 17:12 tbenthompson

Reopened because this is something I want to do eventually.

tbenthompson avatar Jan 07 '21 18:01 tbenthompson