[BUG] darts should provide way to input multiprocessing_context parameter for dataloader when fit

[joshua-xia]

Describe the bug I use Parallel(n_jobs=2, backend="loky") to boot two training job, and the dataloader workers number set to be 3

when training the folliowing error happened:

pred_values_dict_list = Parallel(n_jobs=2, backend="loky")(

File "/devtools/anaconda3/envs/quant-env/lib/python3.9/site-packages/joblib/parallel.py", line 1952, in call return output if self.return_generator else list(output) File "/devtools/anaconda3/envs/quant-env/lib/python3.9/site-packages/joblib/parallel.py", line 1595, in _get_outputs yield from self._retrieve() File "/devtools/anaconda3/envs/quant-env/lib/python3.9/site-packages/joblib/parallel.py", line 1699, in _retrieve self._raise_error_fast() File "/devtools/anaconda3/envs/quant-env/lib/python3.9/site-packages/joblib/parallel.py", line 1734, in _raise_error_fast error_job.get_result(self.timeout) File "/devtools/anaconda3/envs/quant-env/lib/python3.9/site-packages/joblib/parallel.py", line 736, in get_result return self._return_or_raise() File "/devtools/anaconda3/envs/quant-env/lib/python3.9/site-packages/joblib/parallel.py", line 754, in _return_or_raise raise self._result AttributeError: 'Process' object has no attribute 'env'

the reason is when create dataloader, we should input multiprocessing_context=get_context("loky"), but there is no place to setting the multiprocessing_context from neither model parameter or fit parameter.

so I think we should be able to set multiprocessing_context in fit function when we set num_loader_workers>0

[madtoinou]

Hi @joshua-xia,

I think that this problem will be solved by #2295, if you cannot wait for it to be merged and released, you inspire yourself from the changes to apply it to your local darts install.

[joshua-xia]

I did not see the multiprocessing_context from that issue 's patch, I strongly suggest the committer add this parameter.

[dennisbader]

#2295 has been merged now which adds support through dataloader_kwargs when calling fit/predict/...