varianttools icon indicating copy to clipboard operation
varianttools copied to clipboard

Python 3.8 use spawn as default method to create subprocesses.

Open BoPeng opened this issue 5 years ago • 3 comments

Perhaps I have some particular version of SWIG?

> vtools import test.vcf                                                            INFO: Using primary reference genome hg19 of the project.
INFO: Importing variants from test.vcf (1/1)
ERROR: cannot pickle 'SwigPyObject' object
Traceback (most recent call last):
  File "/Users/bpeng/vatlab/vtools/src/variant_tools/importer.py", line 1024, in importVariants
    store.importGenotypes(importer)
  File "/Users/bpeng/vatlab/vtools/src/variant_tools/geno_store.py", line 1855, in importGenotypes
    return importGenotypesInParallel(importer)
  File "/Users/bpeng/vatlab/vtools/src/variant_tools/importer_allele_hdf5.py", line 1517, in importGenotypesInParallel

BoPeng avatar May 11 '20 22:05 BoPeng

Just tested python 3.6, and python 3.7 and they work just fine ... so the problem is only presented in python 3.8. Note that Python 3.8 uses a new protocol for Pickle as default.

BoPeng avatar May 11 '20 22:05 BoPeng

The problem is caused by https://bugs.python.org/issue33725

Long story short: Newer versions of MacOSX does not like fork() and might cause problems. CPython guys understood this and change the default method to fork process with spawn.

Now, for fork we are NOT picking stuff like functions, and on spawn we are, then a lot of things will sudden fall apart.

BoPeng avatar May 12 '20 02:05 BoPeng

This is not a clean fix since fork should be deprecated (at least as suggested by the core team), but it is a quick fix to make vtools work again with python 3.8.

BoPeng avatar May 12 '20 04:05 BoPeng