cppimport
cppimport copied to clipboard
Add configuration for setting the degree of compilation parallelism.
Maybe the first question is how many cppimport will use by default. Since I'm running it on a machine with a small memory, sometimes I can see the compilation goes out of memory.
Currently, parallelism is off by default. If you turn parallelism on, the number of cores is equal to multiprocessing.cpu_count().
This would be quite easy to modify. I have two proposals for how we could do this:
- Change the parallelism flag to instead take a parameter indicating the number of threads to use. Ideally, this would be done in a way that maintains backward compatibility with the current True/False behavior. So, perhaps, True == multiprocessing.cpu_count, False = 1, and then any integer uses that number of threads.
- A slightly different, but similar proposal would be to just add another configuration field that contains the number of threads to use.
Some locations in the code that would be useful for this: https://github.com/tbenthompson/cppimport/blob/00c40d9a7f706409156a42ff3a7934e54a8050c1/cppimport/templating.py#L46 https://github.com/tbenthompson/cppimport/blob/00c40d9a7f706409156a42ff3a7934e54a8050c1/cppimport/build_module.py#L57 https://github.com/tbenthompson/cppimport/blob/00c40d9a7f706409156a42ff3a7934e54a8050c1/cppimport/build_module.py#L150
I would be excited to accept a pull request if you have the time to implement this!
Reopened because this is something I want to do eventually.