qutip-notebooks Parallelism example does not work

Parallelism example does not work

Open yuchiuchao opened this issue 4 years ago • 12 comments

Hello, I am trying to reproduce the results in the example notebook qutip-notebooks/examples/temporal-photon-scattering.ipynb at this location https://github.com/qutip/qutip-notebooks/blob/master/examples/temporal-photon-scattering.ipynb

I could reproduce all results with no problem, except when I got into the parallelized part. The notebook says BUSY, but there is basically no CPU consumed by the notebook and it stays there forever. Could you suggest some troubleshooting ideas for m to further debug this problem. There is no error message produced. Everything looks quiet, except the BUSY indicator and 0 CPU consumption. I checked the worker_count value, which is 7, and I have 8 CPU's. Query for pool returned <multiprocessing.pool.Pool at 0x11cd8f10>

This is the version table:

Software	Version
QuTiP	4.4.1
Numpy	1.17.1
SciPy	1.3.1
matplotlib	3.1.1
Cython	0.29.13
Number of CPUs	8
BLAS Info	OPENBLAS
IPython	7.7.0
Python	3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)]
OS	nt [win32]

Thanks!

Yu Chao Fermi National Accelerator Laboratory

Feb 23 '20 10:02 yuchiuchao

@fminga can you help with this?

Feb 23 '20 11:02 nathanshammah

I ran it in the Jupyter browser mode and captured the following error traceback lines when the parallelized part was executed. Hope this provides a clue:

[I 16:03:14.918 NotebookApp] Kernel started: 2ee5d5e0-49db-4738-9f39-01ece96ba77e [I 16:03:16.781 NotebookApp] Adapting to protocol v5.1 for kernel 2ee5d5e0-49db-4738-9f39-01ece96ba77e Process SpawnPoolWorker-1: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-2: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-6: Process SpawnPoolWorker-4: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-5: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-3: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-7: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-8: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-9: Process SpawnPoolWorker-10: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) Process SpawnPoolWorker-11: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-12: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-13: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-14: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-15: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-16: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-17: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-20: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() Process SpawnPoolWorker-19: Process SpawnPoolWorker-18: File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-21: Traceback (most recent call last): Process SpawnPoolWorker-22: File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) Traceback (most recent call last): AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-23: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-24: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)> Process SpawnPoolWorker-25: Traceback (most recent call last): File "c:\python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "c:\python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "c:\python37-32\lib\multiprocessing\pool.py", line 110, in worker task = get() File "c:\python37-32\lib\multiprocessing\queues.py", line 354, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'scattering_probability_multiprocess' on <module 'main' (built-in)>

Feb 24 '20 22:02 yuchiuchao

Dear Yu,

Regarding this and especially #101: the comment in that issue has a tone that I do not like. Here we all try to help each other. I am glad that issue is closed, but let me address both issues here, as anyone subscribed to the repository got a notification and I do not want it to lead to a toxic environment for the community, as it clearly puts off maintainers and contributors.

tone

First, to the way issues are raised and answered in this community: QuTiP is maintained by a team of volunteers. QuTiP is developed and maintained among all the things the contributors do, as this is generally on top of research or work duties of early-career researchers, mostly students and post docs.

It is a community project and a patch of many people’s contributions to the code base. Some eventually may not work on all users configurations. However, all notebooks are thoroughly tested, as well as the main qutip codebase, with excellent best practices such as independent code review, PEP8, continuous integration tools, automatic syntax readability checks, and with unit tests that cover more than 70% of the lines of code.

When the open-source ecosystem updates or the rest of the code base is changed, it may happen that some things do not work out for all users configurations. However, I think it is unfair to say that anyone who contributes here would "waste" someone's time, as per #101. There are over 60 tutorials and lectures on the website, and most run fine, the others generally with minor bugs. Any help in fixing them is welcome.

Let me also mention that parallel instances are pretty advanced features, so they do not involve beginners expectations.

parallelism

Now, to the code problems: I am sorry that you could not run the notebooks and I hope someone can look into helping solve the problems. If you ask politely, it is more likely that someone will look into this.

I tried to run the temporal-photon-scattering.ipynb and the parallel-JC-steadystate.ipynb notebooks (I expect that is the second notebook you cite) and both ran fine. The first one took a long time in the last cell, but this can be amended by decreasing the lists length. It is using all of the CPU cores on my machine. I suspect something is off with your environment configuration, as many users make extensive use of qutip's parfor, although some may be sending jobs in the form of python scripts. Printing qutip.about() can help in understanding more, as it provides some additional information to the table. Mine is:

QuTiP Version:      4.5.0
Numpy Version:      1.17.1
Scipy Version:      1.2.1
Cython Version:     0.29.8
Matplotlib Version: 3.1.2
Python Version:     3.7.3
Number of CPUs:     2
BLAS Info:          OPENBLAS
OPENMP Installed:   False
INTEL MKL Ext:      False
Platform Info:      Darwin (x86_64)
Installation path:  /miniconda3/lib/python3.7/site-packages/qutip

Feb 27 '20 02:02 nathanshammah

The error in Jupyter notebook is a problem with multiprocessing in notebooks on windows. I had this problem before, and I think the workaround is to put the function scattering_probability_multiprocess() in an external .py file, and import it in the jupyter notebook.

I don't know if this will fix the original problem.

Feb 27 '20 02:02 nwlambert

Hi Neill,

Thank you for the answer! Indeed I tried that after searching for some answers, but then the problem is that all variables defined in the notebook proper become not local to the imported function, including pretty basic ones like the creation operator etc. To make all of them known to this imported function gets unwieldy.

So you mean this multi-processing feature is a problem with the Jupyter notebook?

Thanks!

On 2/26/2020 8:59 PM, Neill Lambert wrote:

The error in Jupyter notebook is a problem with multiprocessing in notebooks on windows. I had this problem before, and I think the workaround is to put the function scattering_probability_multiprocess() in an external .py file, and import it in the jupyter notebook.

I don't know if this will fix the original problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_qutip_qutip-2Dnotebooks_issues_100-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAMQTRBDWMIKKXI2Y5WATLNDRE4UCJA5CNFSM4KZZMFX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENCXHSA-23issuecomment-2D591754184&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=dHtTtJl5sgv50oSLX7yWuhLlTo7Whu4egwNqn0QnWkU&s=3w1M9S3qeOnFo5DnKh_Arr04VFD22vsYsJ0zR-ZyGfg&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMQTRBEWEAMA77L2LNETZBTRE4UCJANCNFSM4KZZMFXQ&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=dHtTtJl5sgv50oSLX7yWuhLlTo7Whu4egwNqn0QnWkU&s=1SA__D2z8rTLvDc8e4Sbw-Nf8ixjDUUwpu-1XVQol70&e=.

Feb 27 '20 04:02 yuchiuchao

Dear Nathan,

It was certainly not my intention to poison the atmosphere of this forum and I apologize if I inadvertently offended anyone's sensitivity. It is really a cultural issue and certainly a lesson learned for me. I came from many years in the environment of accelerator control systems and applications at national labs. Comments from users about codes developed by scientists and engineers, such as what I wrote, would be considered mild. I myself have been the target of such comments for sure. The perception is that, if one puts out a product in public domain and expect to be recognized for it as a user tool, it is expected to run all the time on all platforms by everyone without a glitch. The user is not expected to debug your code. Absent that, one acquires a negative reputation from that point on for a long time, something everyone works hard to avoid. Of course, it was insensitive on my part to apply that metric elsewhere, where the same context may not apply, a good lesson for me. I realize this is not a forum on sociology so I will stop here. Suffice it to say It is incumbent on me to adapt to the culture of the land when I step into other territories, not the other way around.

Thank you for trying out the code. So it remains a puzzle as to what caused my problem. I ran the qutip.about() command and didn't get results too different from yours.

QuTiPVersion:4.3.1NumpyVersion:1.16.4ScipyVersion:1.2.1CythonVersion:0.29.10MatplotlibVersion:3.1.0PythonVersion:3.7.3NumberofCPUs:8BLASInfo:OPENBLASOPENMPInstalled:FalseINTELMKLExt:FalsePlatformInfo:Windows(AMD64)Installationpath:C:\Python37-32\lib\site-packages\qutip

cheers,

On 2/26/2020 8:31 PM, Nathan Shammah wrote:

Dear Yu,

Regarding this and especially #101 https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_qutip_qutip-2Dnotebooks_issues_101&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=JtNJZJeK32ok3Vj_ulnOFaqILZnHFZZDZDUxXaqyVxc&s=tErShMOw9ZR8uomMs0FZ510ThwDCVM3QipbHj8tOUjc&e=: the comment in that issue has a tone that I do not like. Here we all try to help each other. I am glad that issue is closed, but let me address both issues here, as anyone subscribed to the repository got a notification and I do not want it to lead to a toxic environment for the community, as it clearly puts off maintainers and contributors.

First, to the way issues are raised and answered in this community: QuTiP is maintained by a team of volunteers. QuTiP is developed and maintained among all the things the contributors do, as this is generally on top of research or work duties of early-career researchers, mostly students and post docs.

It is a community project and a patch of many people’s contributions to the code base. Some eventually may not work on all users configurations. However, all notebooks are thoroughly tested, as well as the main qutip codebase, with excellent best practices such as independent code review, PEP8, continuous integration tools, automatic syntax readability checks, and with unit tests that cover more than 70% of the lines of code.

When the open-source ecosystem updates or the rest of the code base is changed, it may happen that some things do not work out for all users configurations. However, I think it is unfair to say that anyone who contributes here would "waste" someone's time, as per #101 https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_qutip_qutip-2Dnotebooks_issues_101&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=JtNJZJeK32ok3Vj_ulnOFaqILZnHFZZDZDUxXaqyVxc&s=tErShMOw9ZR8uomMs0FZ510ThwDCVM3QipbHj8tOUjc&e=. There are over 60 tutorials and lectures on the website, and most run fine, the others generally with minor bugs. Any help in fixing them is welcome.

Let me also mention that parallel instances are pretty advanced features, so they do not involve beginners expectations.

Now, to the code problems: I am sorry that you could not run the notebooks and I hope someone can look into helping solve the problems. If you ask politely, it is more likely that someone will look into this.

I tried to run the temporal-photon-scattering.ipynb and the parallel-JC-steadystate.ipynb notebooks (I expect that is the second notebook you cite) and both ran fine. The first one took a long time in the last cell, but this can be amended by decreasing the lists length. It is using all of the CPU cores on my machine. I suspect something is off with your environment configuration, as many users make extensive use of qutip's parfor, although some may be sending jobs in the form of python scripts. Printing |qutip.about()| can help in understanding more, as it provides some additional information to the table. Mine is:

|QuTiP Version: 4.5.0 Numpy Version: 1.17.1 Scipy Version: 1.2.1 Cython Version: 0.29.8 Matplotlib Version: 3.1.2 Python Version: 3.7.3 Number of CPUs: 2 BLAS Info: OPENBLAS OPENMP Installed: False INTEL MKL Ext: False Platform Info: Darwin (x86_64) Installation path: /miniconda3/lib/python3.7/site-packages/qutip |

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_qutip_qutip-2Dnotebooks_issues_100-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAMQTRBD65EKR4MOZJLPJBELRE4QZHA5CNFSM4KZZMFX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENCVSLY-23issuecomment-2D591747375&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=JtNJZJeK32ok3Vj_ulnOFaqILZnHFZZDZDUxXaqyVxc&s=TDR9Y92kAJDpybXtTyXgkCSH3FpUeHAjDTcFoOj8Pcc&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMQTRBEUAKBFYQQ5RZJK4MLRE4QZHANCNFSM4KZZMFXQ&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=JtNJZJeK32ok3Vj_ulnOFaqILZnHFZZDZDUxXaqyVxc&s=HVPkfgZ3XyO7YlMKXIwtvsP88XCvyKje2d1XWr7-4-w&e=.

Feb 27 '20 06:02 yuchiuchao

I do not know the origin, but googling suggests that it is a lack of compatibility between multiprocessing and notebooks in windows.

Your options are to not use notebooks, refactor the code using the above workaround, or try using python under Windows Subsystems for Linux (though this requires a bit of work of installing WSL, ubuntu, python etc).

for the workaround, simply pasting this into an external file:

from qutip import scattering_probability, destroy

def Htls_rft(gamma, pulseLength, pulseArea):
    RabiFreq = pulseArea / (2*pulseLength)
    sm = destroy(2)
    return [[sm.dag() + sm, lambda t, args: RabiFreq * (t < pulseLength)]]

def scattering_probability_multiprocess(pulse_area, n, psi0, sm, tlist, gamma, pulse_length):
    # Helper function to allow pool.map parallelism
    return scattering_probability(Htls_rft(gamma, pulse_length, pulse_area), psi0, n, [sm], tlist)

Importing scattering_probability_multiprocess from the external file, and then calling this works


for n in emission_nums:
    args = [(pulse_area, n, psi0, sm, tlist, gamma, pulse_length) for pulse_area in pulse_areas]
    
    scatter_probs = pool.starmap(scattering_probability_multiprocess, args)
    plt.plot(pulse_areas / np.pi, scatter_probs, label = "$P_{}$".format(n))

Feb 27 '20 06:02 nwlambert

Great! Thanks! Let me give it a try.

On 2/27/2020 12:58 AM, Neill Lambert wrote:

I do not know the origin, but googling suggests that it is a lack of compatibility between multiprocessing and notebooks in windows.

Your options are to not use notebooks, refactor the code using the above workaround, or try using python under Windows Subsystems for Linux (though this requires a bit of work of installing WSL, ubuntu, python etc).

for the workaround, simply pasting this into an external file:

|from qutip import scattering_probability, destroy def Htls_rft(gamma, pulseLength, pulseArea): RabiFreq = pulseArea / (2*pulseLength) sm = destroy(2) return [[sm.dag() + sm, lambda t, args: RabiFreq * (t < pulseLength)]] def scattering_probability_multiprocess(pulse_area, n, psi0, sm, tlist, gamma, pulse_length): # Helper function to allow pool.map parallelism return scattering_probability(Htls_rft(gamma, pulse_length, pulse_area), psi0, n, [sm], tlist) |

Importing scattering_probability_multiprocess from the external file, and then calling this works

|for n in emission_nums: args = [(pulse_area, n, psi0, sm, tlist, gamma, pulse_length) for pulse_area in pulse_areas] scatter_probs = pool.starmap(scattering_probability_multiprocess, args) plt.plot(pulse_areas / np.pi, scatter_probs, label = "$P_{}$".format(n)) |

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_qutip_qutip-2Dnotebooks_issues_100-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAMQTRBEJYSJCMMDW5O4PM6TRE5QCTA5CNFSM4KZZMFX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENDF5MY-23issuecomment-2D591814323&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=-DI3jvvBj4LzIACiReEdOM1sTjvYmPxpU1lJ8pDqkfU&s=RsuGWRVUX0zZZjlV6LM5vnbcRzLVTBG-e-6BDELQgcQ&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMQTRBG2KFKAKCQPAOZDNE3RE5QCTANCNFSM4KZZMFXQ&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=-DI3jvvBj4LzIACiReEdOM1sTjvYmPxpU1lJ8pDqkfU&s=tQVj_zrnQKjUD3lmuZ0afTe98ejICfuVEG5uBUN1eMc&e=.

Feb 27 '20 07:02 yuchiuchao

Hi Neill,

The workaround definitely worked! Thanks a lot!

For the long run I'll consider the Linux route for sure.

cheers,

On 2/27/2020 12:58 AM, Neill Lambert wrote:

I do not know the origin, but googling suggests that it is a lack of compatibility between multiprocessing and notebooks in windows.

Your options are to not use notebooks, refactor the code using the above workaround, or try using python under Windows Subsystems for Linux (though this requires a bit of work of installing WSL, ubuntu, python etc).

for the workaround, simply pasting this into an external file:

|from qutip import scattering_probability, destroy def Htls_rft(gamma, pulseLength, pulseArea): RabiFreq = pulseArea / (2*pulseLength) sm = destroy(2) return [[sm.dag() + sm, lambda t, args: RabiFreq * (t < pulseLength)]] def scattering_probability_multiprocess(pulse_area, n, psi0, sm, tlist, gamma, pulse_length): # Helper function to allow pool.map parallelism return scattering_probability(Htls_rft(gamma, pulse_length, pulse_area), psi0, n, [sm], tlist) |

Importing scattering_probability_multiprocess from the external file, and then calling this works

|for n in emission_nums: args = [(pulse_area, n, psi0, sm, tlist, gamma, pulse_length) for pulse_area in pulse_areas] scatter_probs = pool.starmap(scattering_probability_multiprocess, args) plt.plot(pulse_areas / np.pi, scatter_probs, label = "$P_{}$".format(n)) |

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_qutip_qutip-2Dnotebooks_issues_100-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAMQTRBEJYSJCMMDW5O4PM6TRE5QCTA5CNFSM4KZZMFX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENDF5MY-23issuecomment-2D591814323&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=-DI3jvvBj4LzIACiReEdOM1sTjvYmPxpU1lJ8pDqkfU&s=RsuGWRVUX0zZZjlV6LM5vnbcRzLVTBG-e-6BDELQgcQ&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMQTRBG2KFKAKCQPAOZDNE3RE5QCTANCNFSM4KZZMFXQ&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=-DI3jvvBj4LzIACiReEdOM1sTjvYmPxpU1lJ8pDqkfU&s=tQVj_zrnQKjUD3lmuZ0afTe98ejICfuVEG5uBUN1eMc&e=.

Feb 27 '20 09:02 yuchiuchao

I've run into this problem as well. The root cause is a rather subtle difference in how subprocesses are spawned in Windows vs Linux. As of Python 3.8, the problem also affects macOS. I started to explore a potential solution in https://github.com/qutip/qutip/pull/1092 but didn't have time to finish it. See the tweets linked in that PR for more details on the underlying technical issue. Ultimately, it's that top level functions in notebooks can't be pickled (while functions in scripts/modules can)

My solution was to use a third-party library, loky as a replacement for the standard-library pickle. I did the PR mostly just for testing whether this works (before getting stuck on difficulties running the tests), but I'm pretty sure it would solve the problem. It would be up for debate through whether we want to add loky as an additional dependency for qutip. If so, we could try to actually finish that PR.

Feb 27 '20 15:02 goerz

Also see https://qucontrol.github.io/krotov/v1.0.0/notebooks/08_example_ensemble.html (cell 2), which is where I ran into this.

Feb 27 '20 15:02 goerz

Hi Michael,

Thanks a lot for this input! Look forward to future developments on this.

On 2/27/2020 9:36 AM, Michael Goerz wrote:

I've run into this problem as well. The root cause is a rather subtle difference in how subprocesses are spawned in Windows vs Linux. As of Python 3.8, the problem also affects macOS. I started to explore a potential solution in qutip/qutip#1092 https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_qutip_qutip_pull_1092&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=PJJ_oB5qDUq4ejy_bg25-Fi65DVExYDut7eYyT9gsrw&s=5sky-ljJsjK8RV3lV18ChNugWYmOKJZ5CJoDtznMnQs&e= but didn't have time to finish it. See the tweets linked in that PR for more details on the underlying technical issue. Ultimately, it's that top level functions in notebooks can't be pickled (while functions in scripts/modules can)

My solution was to use a third-party library, loky https://urldefense.proofpoint.com/v2/url?u=https-3A__loky.readthedocs.io_en_stable_&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=PJJ_oB5qDUq4ejy_bg25-Fi65DVExYDut7eYyT9gsrw&s=yKVqHh9NZuOHDYNy5D-1EYcEpDIgoEToHzZogDL0FIM&e= as a replacement for the standard-library pickle. I did the PR mostly just for testing whether this works (before getting stuck on difficulties running the tests), but I'm pretty sure it would solve the problem. It would be up for debate through whether we want to add loky as an additional dependency for qutip. If so, we could try to actually finish that PR.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_qutip_qutip-2Dnotebooks_issues_100-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAMQTRBAI5A7SHNEF5QAEYADRE7MWNA5CNFSM4KZZMFX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENEZDLA-23issuecomment-2D592023980&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=PJJ_oB5qDUq4ejy_bg25-Fi65DVExYDut7eYyT9gsrw&s=6Zvs-x7njm6o-d3VqhzZYlvmifsJu314SI3OoFy2jqk&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMQTRBDSXW3JH4NA44A3MMLRE7MWNANCNFSM4KZZMFXQ&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=5ftqdDun7KKoNX7sQ-96Og&m=PJJ_oB5qDUq4ejy_bg25-Fi65DVExYDut7eYyT9gsrw&s=lSmSeWjfG3GnM343-AMR-8DDtaxe-tENgOx9I4NXUBU&e=.

Feb 27 '20 21:02 yuchiuchao

qutip-notebooks qutip-notebooks copied to clipboard

Parallelism example does not work

tone

parallelism

qutip-notebooks
qutip-notebooks copied to clipboard