qutip
lighter qutip import
- imports should be avoided internally to try and make single qutip module imports more lightweight
Is multiprocessing used by default in some functions ? Can we do away with that in the init.py file and add it where it is used ?
Why do we need to check for numpy and scipy versions at each import ? Is it not enough to check it when installing qutip ?
What do you mean by "avoiding internal imports" ? Do you mean to say that we should remove all * imports ?
from qutip.tomography import *
multiprocessing is used in __init__ to determine how many cpus are available. I guess you are right in that it is unlikely that the number of cpus would change between install and use.
So at the moment when we do something simple like
from qutip import Qobj
All the * imports in __init__ are called. That is a lot of functions now.
What we have to be careful of though is that a lot of users start off their qutip session with
from qutip import *
Now it seems reasonable that they get everything, as it is now. But that for users who are more careful, like the first example, should only get what they ask for. Now importing Qobj gets you a lot, as it uses a lot of qutip functions, but not all the solvers for instance.
@nonhermitian mentioned something about being able to move the pyximport bit to somewhere in the dynamic C building workflow. Now that would be a real bonus. Say we have some (probably Windows) users who have not managed to configure a working C compiler, but have installed from a conda package or some other wheel like thing, then they would be able to use everything except the string format time-dependent stuff. This is also discussed in #448
Ok so if we have to start we do away with all the * imports first except
from qutip.qobj import *
then we can remove the numpy and scipy checks but numpy is still used to setup pyximport in
# Setup pyximport
import pyximport
os.environ['CFLAGS'] = '-O2 -w -ffast-math'
pyximport.install(setup_args={'include_dirs': [numpy.get_include()]})
del pyximport
Then we can move the section regarding multiprocessing to some appropiate place. Currently it is used in the following files. Can we put the multiprocessing stuff in a file called multiprocessing_config.py and run it to get the settings in __init__ at all the places where we need multiprocessing.
/Users/shahnawaz/dev/qutip/qutip/__init__.py:
33 from __future__ import division, print_function, absolute_import
34 import os
35: # Fix the multiprocessing issue with NumPy compiled against OPENBLAS
36 if 'OPENBLAS_MAIN_FREE' not in os.environ:
37 os.environ['OPENBLAS_MAIN_FREE'] = '1'
..
149 # cpu/process configuration
150 #
151: import multiprocessing
152
153 # Check if environ flag for qutip processes is set
...
164 qutip.settings.num_cpus = info['cpus']
165 else:
166: qutip.settings.num_cpus = multiprocessing.cpu_count()
167
168
...
270 # Clean name space
271 #
272: del os, sys, numpy, scipy, multiprocessing
/Users/shahnawaz/dev/qutip/qutip/fortran/mcsolve_f90.py:
83 serial : boolean
84 If True (default is False) the solver will not make use of the
85: multiprocessing module, and simply run in serial.
86 ptrace_sel: list
87 This optional argument specifies a list of components to keep when
..
226
227 def parallel(self):
228: from multiprocessing import Process, Queue, JoinableQueue
229
230 if debug:
/Users/shahnawaz/dev/qutip/qutip/hardware_info.py:
36 import os
37 import sys
38: import multiprocessing
39
40 def _mac_hardware_info():
..
91 ncpus += int(cpu.Properties_['NumberOfCores'].Value)
92 except:
93: ncpus = int(multiprocessing.cpu_count())
94 return {'os': 'Windows', 'cpus': ncpus}
95
/Users/shahnawaz/dev/qutip/qutip/mcsolve.py:
225 if config.options.num_cpus == 1:
226 # fallback on serial_map if num_cpu == 1, since there is no
227: # benefit of starting multiprocessing in this case
228 config.map_func = serial_map
229
...
264 time_type, h_stuff, c_stuff = _td_format_check(H, c_ops, 'mc')
265 c_terms = len(c_stuff[0]) + len(c_stuff[1]) + len(c_stuff[2])
266: # set time_type for use in multiprocessing
267 config.tflag = time_type
268
/Users/shahnawaz/dev/qutip/qutip/parallel.py:
33 """
34 This function provides functions for parallel execution of loops and function
35: mappings, using the builtin Python module multiprocessing.
36 """
37 __all__ = ['parfor', 'parallel_map', 'serial_map']
38
39 from scipy import array
40: from multiprocessing import Pool
41 from functools import partial
42 import os
/Users/shahnawaz/dev/qutip/qutip/settings.py:
32 ###############################################################################
33 """
34: This module contains settings for the QuTiP graphics, multiprocessing, and
35 tidyup functionality, etc.
36 """
Can the pyximport be dealt in a similar way by having a pyximport_config.py and running it from all the places which needs C compilation.
@ajgpitch @nonhermitian In ProjectQ they have an install option which lets you use the library without the C++ simulator by specifying the parameter --without-cppsimulator during a python setup.py install Link. We can try something like that if you want a version of qutip which can run even without a working compiler. Here is their setup.py file with the conditional compilation. In any case moving the .pyx files so that user has the option to run qutip even without a working compiler setup seems like a good idea.
But again, we use runtime code generation for the time-dependent stuff. Cython is in every Python dist, so I see little in the way of having that as a requirement. Getting a compiler is also trivial on every supposed platform.
We have an install now, through conda-forge packages, and #570, that works on all platforms, even without a C compiler. We have a working compiler solution for all platforms except Python 3.5/3.6 on Windows. So I don't think we need to work further on making qutip work without a C compiler. We should be trying to include the dynamic C for all users, which we almost have.
I am keen on making the import lighter though. Though it's not super urgent. I will be looking more into the pyximport and settings soon. I think we should be able to move the pyximport.install to just before when it is needed. And the same with the hardware_info call.
I've moved this to the 5.0 milestone. I'm considering migrating QuTiP 5 to https://scientific-python.org/specs/spec-0001/, but regardless of how we want to address imports, QuTiP 5 is the time to do it.
The QuTiP 4 and 5 imports were made quite a bit faster by fixing how we set up loggers (or removing them).
