pynwb
pynwb copied to clipboard
NeurodataWithoutBorders
import pynwb is slow
Importing pynwb is surprisingly slow on my machine. Is this typical for others?
Python 3.9.9 (main, Jan 13 2022, 00:56:30)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: %time import pynwb
CPU times: user 2.74 s, sys: 1.7 s, total: 4.44 s
Wall time: 3.52 s
I think all my other packages are loading reasonably fast. For example, h5py loads in ~400 ms.
Python 3.9.9 (main, Jan 13 2022, 00:56:30)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: %time import h5py
CPU times: user 598 ms, sys: 1.47 s, total: 2.06 s
Wall time: 393 ms
@mavaylon1 it would be helpful to run this through a profiler to see where the time is being spent. I assume the issue is likely that HDMF has to construct the TypeMap from the schema on import, but it will be good as a first step to get a clear understanding of where time is being spent so we can see what may be feasible to do to improve time for importing pynwb.
Yes I think importing HDMF accounts for a good chunk of this. Thanks for looking into it.
Running this through a profiler, hdmf accounts for roughly 46-50% of the time for importing pynwb. Within the time spent for hdmf, a very small portion is spent on backend, i.e objectmapper, builders, and manager.
Using a more detailed profiler seems to point to loading the namespaces as a large factor in cumulative time spent.
Related to https://github.com/NeurodataWithoutBorders/pynwb/pull/1050 and new issue https://github.com/NeurodataWithoutBorders/pynwb/issues/1255 . Initially, the problem with the linked approach of pickling the TypeMap is that it was not compatible with loading NWB extensions from a file. But I think this is not a completely blocking -- I think we can still pickle the TypeMap for the core namespace and load NWB extensions from a file. @mavaylon1 can you look into this?
