Support using only a subset of modules

[perlman]

Thanks for this great library for interfacing with CATMAID.

It would be great if I could include a single module (e.g. pymaid.fetch) without the other modules. This is currently not possible due to pymaid/__init__.py automatically importing all of the submodules.

The full set of modules can take a while to load on my machine, and the initialization of VisPy leads to the spawning of a GUI application even if no visualization routine is called.

[schlegelp]

Hi Eric,

Glad you find the package useful!

The namespace issue has been in a thorn in my side for a while. I would love to have the flat namespace (i.e. importing modules in pymaid/init.py to make them available at the package level) while also being able to import modules individually. AFAIK there is no way to have the cake and eat it too but if you have a suggestion, please let me know. Currently, I think the flat namespace is more important for most users.

There are a few things I can try to help though:

1. I will look into optimising startup time (e.g. by moving imports of some rarely used libraries to runtime). Out of curiosity: How long does the import of pymaid take on your machine?
2. For the vispy GUI issue, I see two possibilities: I could either move the vispy import to runtime as well (probably easiest) or implement some sort of headless mode which prevents vispy from being imported and can be activated via e.g. a global variable.

If this is urgent, you could also fork the repository and simply delete the contents of pymaid/init.py. This would allow you to skip a few submodules when importing pymaid.fetch (will still import some dependencies from other pymaid modules though). I just ran a quick test:

With __init__.py as is:

zoopc651:~ philipps$ time python3 -c "from pymaid import fetch"
real	0m2.208s
user	0m1.783s
sys	0m0.403s
zoopc651:~ philipps$ 

With empty __init__.py:

zoopc651:~ philipps$ time python3 -c "from pymaid import fetch"

real	0m1.170s
user	0m0.886s
sys	0m0.267s

So this seems to cut the import time almost in half. Maybe I can squeeze out a bit more by improving imports. I'll keep you posted.

[perlman]

I see nearly identical import time

__init__.py as is:

real	0m2.770s
user	0m2.497s
sys	0m0.461s

With empty __init__.py:

real	0m1.217s
user	0m1.169s
sys	0m0.259s

The times were a lot worse due to an issue with matplotlib and my font cache, but I seem to have that resolved.

All of your suggestions sound good. I have not dug in enough to figure out which bit of code triggers the "app launch" (OS X). It has to be more than a simple import vispy -- maybe a viewer is being initialized at import in one of the modules?

[schlegelp]

Nice.

Re the GUI being initialised: The only vispy-related things that happen at import time are these (across two different modules):

import vispy
from vispy import scene
from vispy.geometry import create_sphere
from vispy.gloo.util import _screenshot

vispy.use(app='PyQt5')

Does any of this spawn a GUI on your system?

[perlman]

None of those do it.

But import pymaid does.

Hmm.

[schlegelp]

Maybe it's matplotlib then? Try these import statements:

import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.lines as mlines
import matplotlib.patches as mpatches
import mpl_toolkits
from mpl_toolkits.mplot3d.art3d import Line3DCollection
from mpl_toolkits.mplot3d import proj3d
from matplotlib.collections import LineCollection
import matplotlib.colors as mcl

[perlman]

Yes, it's matplotlib. I feel bad for trying to place the blame on VisPy.

Specifically, it's ``matplotlib.pyplot'', and is where 0m0.647s of the startup time goes. :-)

[schlegelp]

In that case this should do the trick for you: If you don't need plotting simply import matplotlib in headless mode before importing pymaid:

import matplotlib as mpl
mpl.use('template')

import pymaid

To change the default backend permanently, have a look at your matplotlibrc file.

[perlman]

Thanks! While the startup is still a bit slow, I'm quite content to no longer see the gui launcher. 👍

[schlegelp]

I tried optimizing imports in c7cd8478c2eb88dd478c82fa37819c81694237b3. Unfortunately, there wasn't much to optimize in the first place, so I think any difference will be hardly noticeable.

There are two more options:

1. Decouple pymaid.fetch by creating a "buffer" module of sorts that takes care of higher functions (e.g. the conversion to CatmaidNeuron/List) so that fetch does not have any other pymaid-internal imports. For reference: non-pymaid imports in fetch take 2.2s on my machine, all of pymaid takes 2.3s. Removal of networkx brings fetch down to 1s.
2. Split the whole package into two separate packages, like nat and rcatmaid: one for general neuroanatomy and one for Catmaid interface

The first one option would obviously be easier in the short term, the other one might (a) make the package more interesting for a general audience and (b) be easier to maintain in the long run.

I'll have a think....

[schlegelp]

Came across lazy-import package. Might be worth using this for e.g. vispy and matplotlib

Edit: demand-import also looks good