varianttools
varianttools copied to clipboard
Python 3.8 use spawn as default method to create subprocesses.
Perhaps I have some particular version of SWIG?
> vtools import test.vcf INFO: Using primary reference genome hg19 of the project.
INFO: Importing variants from test.vcf (1/1)
ERROR: cannot pickle 'SwigPyObject' object
Traceback (most recent call last):
File "/Users/bpeng/vatlab/vtools/src/variant_tools/importer.py", line 1024, in importVariants
store.importGenotypes(importer)
File "/Users/bpeng/vatlab/vtools/src/variant_tools/geno_store.py", line 1855, in importGenotypes
return importGenotypesInParallel(importer)
File "/Users/bpeng/vatlab/vtools/src/variant_tools/importer_allele_hdf5.py", line 1517, in importGenotypesInParallel
Just tested python 3.6, and python 3.7 and they work just fine ... so the problem is only presented in python 3.8. Note that Python 3.8 uses a new protocol for Pickle as default.
The problem is caused by https://bugs.python.org/issue33725
Long story short: Newer versions of MacOSX does not like fork() and might cause problems. CPython guys understood this and change the default method to fork process with spawn.
Now, for fork we are NOT picking stuff like functions, and on spawn we are, then a lot of things will sudden fall apart.
This is not a clean fix since fork should be deprecated (at least as suggested by the core team), but it is a quick fix to make vtools work again with python 3.8.