Not working well with multiprocessing library fork start method

[Aspire1Inspire2]

Hi, when I use multiprocessing library on Unix machine, the CAS connection cannot return or print out my evaluated table to my disk, whether I use to_csv() or to_pickle().

However, if I change the start mode to Windows default spawn mode, it will work without issue. multiprocessing.set_start_method('spawn')

Please refer to this link for more information.

[kesmit13]

Are you using the binary or HTTP method of connecting to CAS? I do know that there are some issues when the C extensions for binary connections with Python threads and multi-processing. If you are using binary, you could try HTTP instead and see if that helps.

[Aspire1Inspire2]

We are using binary connection. We found that the http protocol + fork threading is slower than the binary protocol + spawn threading.

It took us a lot of labor hours to figure out that this is not well-supported. If this is a known issue, could we document this in the github.io documentation to avoid future users falling into the same frustrating debugging process?

[kesmit13]

Let me do some digging to see if I can get more information about why it doesn't work, but I think this may be the key (from the link you sent above):

Note that safely forking a multithreaded process is problematic.

The SWAT C extension does use threading, and this may be why it doesn't work properly.