Switch from ete3 to ete4

[dengzq1234]

PR Checklist

• [X] Referenced issue is linked
• [X] If you've fixed a bug or added code that should be tested, add tests!
• [ ] Documentation in docs is updated

Description of changes

This PR is the response to the issue https://github.com/etetoolkit/ete/issues/739, which aim to upgrade the usage of etetoolkit in pertpy from ete3 to ete4.

perpty implement ete for the following tasks: a) browsing trees, such as getting leaves, descendants, ancestors, etc. b) annotation of trees, such as adding information to nodes. c) drawing tree, including drawing tree with layouts of nodes' information.

This repo upgrade the syntax in regards to the first two tasks. The third task which include the drawing part will mostly maintain the usage of ete3 version expect for calling the module in new places

Technical details This PR mostly just correct the syntax of calling the methods or features from ete3 to ete4, straight forward and trivial. detailed differences can be found from

• migration noteos: https://github.com/etetoolkit/ete/wiki/3to4
• ete4 official tutorial https://etetoolkit.github.io/ete/tutorial/tutorial_trees.html#browsing-trees-traversing

I used two ways to conduct the test

1. Passed the unitest this folder /home/deng/Projects/metatree_drawer/test_pertpy/pertpy/tests/tools/_coda

2. Be able to recreate tree images from the step-by-step tutorial of tasccoda tree-aggregated analysis from https://pertpy.readthedocs.io/en/latest/tutorials/notebooks/tasccoda.html

Additional context ete4 installation, ete4 currently is released on githup repo as beta version therefore not yet available in pip nor conda. Users can install ete4 via githup pip install https://github.com/etetoolkit/ete/archive/ete4.zip

In regards to c) drawing tree, ete4 still contains the treeview inherited from ete3 which relies on novel PyQt6 module to render tree image. Users can install the depencies by pip install PyQt6 or install ete4 via pip install -e .[treeview] for a local editable installation.

Important information: In ete4 we will stop maintaining nor making new updates in the old treeview module which is used in pertpy, at this moment it is still functional.

Instead ete4 embrace the new smartview module which has abilities to visualize large tree dataset up to thousands of tips interactively and introduce with novel layouts. The new drawing engine has been implemented in some tools such as PhyloCloud https://phylocloud.cgmlab.org/ (https://doi.org/10.1093/nar/gkac324) TreeProfiler https://doi.org/10.1101/2023.09.21.558621

If pertpy has the demands to visualize trees with ete4 smartview, I'm more than welcome to provide a novel PR in regards to this feature.

[Zethson]

Thank you very much!

ete4 installation, ete4 currently is released on githup repo as beta version therefore not yet available in pip nor conda. Users can install ete4 via github pip install https://github.com/etetoolkit/ete/archive/ete4.zip

This is why I was asking for a release of ete4 on pypi. Currently, I cannot merge/accept this PR because I could not distribute pertpy on pypi. Github dependencies are not supported on pypi for obvious reasons.

What is the status of a pypi release of ete4?

Thank you again!

[Zethson]

Important information: In ete4 we will stop maintaining nor making new updates in the old treeview module which is used in pertpy, at this moment it is still functional.

Instead ete4 embrace the new smartview module which has abilities to visualize large tree dataset up to thousands of tips interactively and introduce with novel layouts. The new drawing engine has been implemented in some tools such as PhyloCloud https://phylocloud.cgmlab.org/ (https://doi.org/10.1093/nar/gkac324) TreeProfiler https://doi.org/10.1101/2023.09.21.558621

If pertpy has the demands to visualize trees with ete4 smartview, I'm more than welcome to provide a novel PR in regards to this feature.

@johannesostner could you please comment on this?

[johannesostner]

Hi @dengzq1234, thanks for helping us with porting our code from ete3 to ete4! Regarding the drawing engine, most of our tree structures are quite small (up to 100 tips), so the performance boost from smartview is not that important for us. Nevertheless, I think that replacing treeview with smartview would be preferable to ensure future compatibility.

Some questions:

• How well does smartview integrate with jupyter notebooks? I think most of our users would like to see the plots rendered directly below their code cells
• Do you know a better way to create the last plot from https://pertpy.readthedocs.io/en/latest/tutorials/notebooks/tasccoda.html (tree with a barplot for every leaf?) Currently, the implmentation for this is quite hacky (save the tree as png and import it into a matplotlib plot).

[dengzq1234]

Thank you very much!

ete4 installation, ete4 currently is released on githup repo as beta version therefore not yet available in pip nor conda. Users can install ete4 via github pip install https://github.com/etetoolkit/ete/archive/ete4.zip

This is why I was asking for a release of ete4 on pypi. Currently, I cannot merge/accept this PR because I could not distribute pertpy on pypi. Github dependencies are not supported on pypi for obvious reasons.

What is the status of a pypi release of ete4?

Thank you again!

I see the issue, the current ete4 is undergoing development on one of the major features which is not showing on the github yet therefore we haven't put it on pypi. I will need to coordinate to see if we want to release the beta version on pypi or wait until it is fully ready. I will keep you updated!

[dengzq1234]

Hi @dengzq1234, thanks for helping us with porting our code from ete3 to ete4! Regarding the drawing engine, most of our tree structures are quite small (up to 100 tips), so the performance boost from smartview is not that important for us. Nevertheless, I think that replacing treeview with smartview would be preferable to ensure future compatibility.

Some questions:

• How well does smartview integrate with jupyter notebooks? I think most of our users would like to see the plots rendered directly below their code cells
• Do you know a better way to create the last plot from https://pertpy.readthedocs.io/en/latest/tutorials/notebooks/tasccoda.html (tree with a barplot for every leaf?) Currently, the implmentation for this is quite hacky (save the tree as png and import it into a matplotlib plot).

Hi! If the tree is relatively small then I think treeview at this moment will handle just fine and tree will be shown clearly.

To your questions:

1. Here is how it goes in Jupyter notebook, it provide a interactive interface so users can manipulate the tree.

2. I have seen the barplot drawing in the function, and yes and no for current ete4. Yes we can draw the barplot for every leaf, such as

And no because your barplot is bidirectional barplot with negative, that's something we are currently working on it.

One of the downside of current ete4 is instead of drawing a tree as static image, we open the tree as a browsing and interactive session, users need to active this browsing session in order to output the static png/pdf for tree. For users who need to automate the tree drawing pipeline, it might be a hinder. But as you said if your users would like to see the tree in the jupyter cells, then I would say ete4 smartview fit this purpose.

[Zethson]

@dengzq1234 kindly pinging you after 3 months:

I see the issue, the current ete4 is undergoing development on one of the major features which is not showing on the github yet therefore we haven't put it on pypi. I will need to coordinate to see if we want to release the beta version on pypi or wait until it is fully ready. I will keep you updated!

Do you have an update on this?

[Zethson]

@dengzq1234 sorry to ping you again, but is there a status update? I don't want to be annoying but I'd be happy to adopt ete4

[dengzq1234]

Hi @Zethson, i am very sorry for the late reply. I did see the message but I don't have answer in regards to the status update. Now it's more of a decision making about if we release ete4 to pip in current state or wait until a major feature. I'm not able to give you the answer because I am also waiting for the answer. :(

[Zethson]

@dengzq1234 I just saw that ete4 is finally on pypi. Would you kindly be willing to revisit this PR so that we can get it in now? I would be very happy and grateful.

[dengzq1234]

@dengzq1234 I just saw that ete4 is finally on pypi. Would you kindly be willing to revisit this PR so that we can get it in now? I would be very happy and grateful.

Hi Lukas, yes I was about to do it. Should be finished by next week.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 3.03030% with 32 lines in your changes missing coverage. Please review.

Project coverage is 56.10%. Comparing base (72c32a2) to head (b5b0f42).

Files with missing lines	Patch %	Lines
pertpy/tools/_coda/_base_coda.py	0.00%	26 Missing :warning:
pertpy/tools/_coda/_tasccoda.py	0.00%	6 Missing :warning:

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #618   +/-   ##
=======================================
  Coverage   56.10%   56.10%           
=======================================
  Files          47       47           
  Lines        6149     6149           
=======================================
  Hits         3450     3450           
  Misses       2699     2699           

Files with missing lines	Coverage Δ
pertpy/tools/__init__.py	97.14% <100.00%> (ø)
pertpy/tools/_coda/_tasccoda.py	0.00% <0.00%> (ø)
pertpy/tools/_coda/_base_coda.py	0.00% <0.00%> (ø)

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dengzq1234]

hi @Zethson, I reviewed the PR and reran and recreated the analysis from the above steps.

• Passed the unitest this folder /home/deng/Projects/metatree_drawer/test_pertpy/pertpy/tests/tools/_coda

• Be able to recreate tree images from the step-by-step tutorial of tasccoda tree-aggregated analysis from https://pertpy.readthedocs.io/en/latest/tutorials/notebooks/tasccoda.html

Here is how to install ete4 for your repo, it requires ete4 and pyqt6. Either install them seperately from pypi in dependencies, or install it as pip install 'ete4[treeview]'.

I think this will complete the tasks that you tool needs from ete. As for the updated visualization for bidirectional barplot as @johannesostner metioned, it's not available yet. I will updated to you once we have this feature.

Thanks for your patience sorry for the inconvenience over this period, and let me know if you need anything else. :)

Best, Ziqi

[Zethson]

Thank you very much! I'll have a look ASAP and will likely take it from here. Is pyqt6 really required? I was hoping that we could drop the big QT dependency.

[dengzq1234]

Thank you very much! I'll have a look ASAP and will likely take it from here. Is pyqt6 really required? I was hoping that we could drop the big QT dependency.

Droping QT dependency was one of our propose to develop ete4, I can try to recreate the layouts and style from your work there. the basic plot_draw_tree() and plot_draw_effects() are easy and I can already do it. The tricky part is the bidirectional barplot for the leaf effect where you used a hacky way to do it. In the current ete4 visualization we only have one directional barplot, for this I need to try something new and maybe talk to the engineer. The difficult part is visualization in ete4 is more dynamic and interactive so it requires longer test in order not to break other things.

[Zethson]

Got it. Consider this a formal feature request then. In any case, this should not stop this PR and I'll get it in regardless. Being able to drop QT was my biggest desire though for this PR.

[dengzq1234]

Got it. Consider this a formal feature request then. In any case, this should not stop this PR and I'll get it in regardless. Being able to drop QT was my biggest desire though for this PR.

Noted and thanks for the feedback! https://github.com/etetoolkit/ete/issues/776. Keep in touch!