jeetsukumaran
Seg Fault when deepcopy-ing large tree
Hello,
I have encountered a Segmentation Fault when attempting to copy a tree. Just reading in the tree and copying it in either of two ways triggers a Seg Fault. The tree has 41,025 taxa and the branch lengths are double-precision float (RAxML output), so it's not small but I've never had a problem like this before. Importing this tree without increasing the recursion limit triggers a different error also, although that is an unrelated and I think previously documented issue (at the higher limit it is fine, until the copy). The code to reproduce it is pretty straightforward:
import sys, dendropy
sys.setrecursionlimit(50000)
tr=dendropy.Tree.get(path=<tree_path>,schema='newick',preserve_underscores=True)
tr2=dendropy.Tree(tr) # --> Segmentation Fault
# ...alternatively...
import copy
tr2=copy.deepcopy(tr) # --> Segmentation Fault
I've attached a zipped text file containing the tree. The details of the environment are below, but in short it's Python 3.7.0 / CentOS Linux 7.5 / DendroPy 4.4.0 (same error with 4.3.0 before updating, fwiw).
Google gives a precious few old hits for Segmentation Faults in Python related to deepcopy() operations, but unfortunately nothing with a helpful solution (sort of like this situation). I know copy and deepcopy can have problems when there are too many recursions, though why that should trigger a seg-fault and why it should happen in this particular instance is not clear. Anyway, I just thought you should know.
Just for fun, I tried the same operation on my laptop using both Windows command prompt and the Ubuntu subsystem. The Windows version couldn't even read the tree without being killed (though it did not say seg-fault). The Ubuntu subsytem actually did not have a problem at all. So go figure... Details and output from those are below also. Please let me know if I can do anything to help and thanks for all you do to make this software available.
Tree: dendropy_segfault_large_tree.zip
Research Sever (CentOS):
*System Specs*
platform :Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
linux_distribution :('CentOS Linux', '7.5.1804', 'Core')
release :3.10.0-862.14.4.el7.x86_64
python_implementation :CPython
python_version :3.7.0
python_build :('default', 'Jul 30 2018 16:46:16')
python_compiler :GCC 7.2.0
libc_ver :('glibc', '2.2.5')
*DendroPy Specs*
DendroPy version : DendroPy 4.4.0 ()
DendroPy location : /usr/local/python/3.7.0/lib/python3.7/site-packages/dendropy
Python version : 3.7.0 (default, Jul 30 2018, 16:46:16) [GCC 7.2.0]
Python executable : /usr/local/python/3.7.0/bin/python3
Python site packages : ['/usr/local/python/3.7.0/lib/python3.7/site-packages']
*Hardware Specs*
model name : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
memory : 256 Tb
stepping : 4
microcode : 0x42d
cpu MHz : 2500.152
cache size : 25600 KB
cpu family : 6
model : 62
Laptop (ASUS, Intel 7th gen chip, 16 Gb RAM)
Windows 10:
C:\Users\miken\Dropbox\GradSchool\Phylogenetics\work\tippstar\tipp2_2019>python
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, dendropy
>>> sys.setrecursionlimit(50000)
>>> treepath='dendropy_segfault_large_tree.tree'
>>> tr=dendropy.Tree.get(path=treepath,schema='newick',preserve_underscores=True)
C:\Users\miken\Dropbox\GradSchool\Phylogenetics\work\tippstar\tipp2_2019>
# *DendroPy Specs*
DendroPy version : DendroPy 4.3.0
DendroPy location : C:\Program Files\Python36\lib\site-packages\dendropy
Python version : 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)]
Python executable : C:\Program Files\Python36\python.exe
Python site packages : ['C:\\Program Files\\Python36', 'C:\\Program Files\\Python36\\lib\\site-packages']
Ubuntu App (from Windows Store):
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, dendropy
>>> sys.setrecursionlimit(50000)
>>> treepath='dendropy_segfault_large_tree.tree'
>>> tr=dendropy.Tree.get(path=treepath,schema='newick',preserve_underscores=True)
>>> tr2=dendropy.Tree(tr)
>>> import copy
>>> tr3=copy.deepcopy(tr)
>>>
# *DendroPy Specs*
DendroPy version : DendroPy 4.4.0
DendroPy location : /usr/local/lib/python3.5/dist-packages/dendropy
Python version : 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]
Python executable : /usr/bin/python3
Python site packages : ['/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.5/dist-packages']
Thank you for reporting this. I can replicate this issue on my end.
By adding a diagnostic ``print'' just before this line:
https://github.com/jeetsukumaran/DendroPy/blob/development-master/src/dendropy/datamodel/basemodel.py#L785
(which is where the fault occurs), it seems that the point of failure is unstable --- it's always a deepcopy of a Node, but sometimes it fails when processing an _edge attribute (which is an object of Edge class) and other times it is when processing an _child_nodes attribute (which is a primitive list object). This discrepancy, and, in particular, that the failure occurs with primitive Python objects as well, coupled with the fact that it seems to work with the WSL Ubuntu (though not on Windows itself --- CRAAAAZY!) leads to me guess that the root cause is deep in the black magic voodoo of the PVM model implementation. But, TBH, I am not sure.
As a work around, would it be possible to simply read the tree twice, passing in the same taxon namespace? There will be a performance hit, of course.
Another work around might be a custom roll-your-own shallow copy, where you iterate over the tree and construct another one in parallel?
fwiw, I think building a python interpreter with symbols turned on, and then running this through valgrind would probably make the pain point obvious. but it would be quite slow.
Thank you both for having a look at this and for the suggestions. The truth is that I'm not sure why a second tree was being created anyway and the biggest lesson for me is that the codebase I'm working with is getting long in the tooth while the use cases have expanded considerably, so an overhaul is in order. Thank you again for your help!
