jwst icon indicating copy to clipboard operation
jwst copied to clipboard

Multiprocessing Start Method Forced to Forkserver

Open TheSkyentist opened this issue 1 year ago • 0 comments

#8093 Forces the multiprocessing start method globally: https://github.com/spacetelescope/jwst/blob/613383f5968b90cc52274682dfd1ead93bddec54/jwst/ramp_fitting/ramp_fit_step.py#L26 https://github.com/spacetelescope/jwst/blob/613383f5968b90cc52274682dfd1ead93bddec54/jwst/jump/jump_step.py#L11

This can cause weird/unexpected behavior on Linux systems (default method fork) vs macOS (default method spawn). Therefore, even if the JWST pipeline is simply imported, it can change the behaviour of unrelated code that wraps the pipeline (this has been my experience). In general, the best practices recommended by Python are to only call the set_start_method within an if __name__ == '__main__' clause.

This may even effect other modules where multiprocessing is called (e.g. wfss_contam) and cause unexpected behaviour. I'm not sure what the best solution is moving forward, but it should be at least reflected in the documentation that this is required under the hood and may explain errors/problems.

TheSkyentist avatar Feb 23 '24 08:02 TheSkyentist