nipype
nipype copied to clipboard
Published
20 hours ago •
nipy
nipy
Importing ``nipype.config`` or ``nipype.logging`` automatically allocates 4.7GB of VMS
Efforts in MRIQC to minimize the baseline process size were resulting in meager gains until I found this. Delaying the import of either module allows reducing the baseline drastically.
To avoid breaking the API, we may need a module proxy that introduces submodules into the namespace without actually loading them. I'm not sure if there's a version that plays nicely with autocomplete tools.
We might be able to get away delaying loading from nipype in MRIQC, except to create the executor plugin.
from nipype.pipeline.plugins.multiproc import MultiProcPlugin adds the 4.7GB too.
I'm going to copy the plugin into MRIQC and see if a bare version of it is more lightweight.
Running improf, it loks like, while nipype is fairly promiscuous about its loads on import, the majority can be explained by numpy and traits (which imports numpy) and cannot plausibly be reduced, loading nipype.pipeline.plugins is inordinately heavy:
$ improf numpy traits,traits.api nipype nipype.pipeline.plugins
numpy: 123 modules; 90.78ms, 365.9MiB
traits: 66 modules; 28.77ms, 4.8MiB
nipype: 112 modules; 40.56ms, -19.1MiB
nipype.pipeline.plugins: 928 modules; 386.15ms, 410.4MiB
Final: 1229 modules; 546.25ms, 762.0MiB
Modules loaded with nipype.pipeline.plugins:
{'IPython',
'IPython.core',
'IPython.core.alias',
'IPython.core.application',
'IPython.core.async_helpers',
'IPython.core.autocall',
'IPython.core.builtin_trap',
'IPython.core.compilerop',
'IPython.core.completer',
'IPython.core.crashhandler',
'IPython.core.debugger',
'IPython.core.display',
'IPython.core.display_functions',
'IPython.core.display_trap',
'IPython.core.displayhook',
'IPython.core.displaypub',
'IPython.core.error',
'IPython.core.events',
'IPython.core.excolors',
'IPython.core.extensions',
'IPython.core.formatters',
'IPython.core.getipython',
'IPython.core.history',
'IPython.core.hooks',
'IPython.core.inputtransformer2',
'IPython.core.interactiveshell',
'IPython.core.latex_symbols',
'IPython.core.logger',
'IPython.core.macro',
'IPython.core.magic',
'IPython.core.magic_arguments',
'IPython.core.magics',
'IPython.core.magics.auto',
'IPython.core.magics.basic',
'IPython.core.magics.code',
'IPython.core.magics.config',
'IPython.core.magics.display',
'IPython.core.magics.execution',
'IPython.core.magics.extension',
'IPython.core.magics.history',
'IPython.core.magics.logging',
'IPython.core.magics.namespace',
'IPython.core.magics.osm',
'IPython.core.magics.packaging',
'IPython.core.magics.pylab',
'IPython.core.magics.script',
'IPython.core.oinspect',
'IPython.core.page',
'IPython.core.payload',
'IPython.core.prefilter',
'IPython.core.profiledir',
'IPython.core.pylabtools',
'IPython.core.release',
'IPython.core.shellapp',
'IPython.core.splitinput',
'IPython.core.ultratb',
'IPython.core.usage',
'IPython.display',
'IPython.extensions',
'IPython.extensions.storemagic',
'IPython.lib',
'IPython.lib.clipboard',
'IPython.lib.display',
'IPython.lib.pretty',
'IPython.paths',
'IPython.terminal',
'IPython.terminal.debugger',
'IPython.terminal.embed',
'IPython.terminal.interactiveshell',
'IPython.terminal.ipapp',
'IPython.terminal.magics',
'IPython.terminal.prompts',
'IPython.terminal.pt_inputhooks',
'IPython.terminal.ptutils',
'IPython.terminal.shortcuts',
'IPython.testing',
'IPython.testing.skipdoctest',
'IPython.utils',
'IPython.utils.PyColorize',
'IPython.utils._process_common',
'IPython.utils._process_posix',
'IPython.utils._sysinfo',
'IPython.utils.capture',
'IPython.utils.colorable',
'IPython.utils.coloransi',
'IPython.utils.contexts',
'IPython.utils.data',
'IPython.utils.decorators',
'IPython.utils.dir2',
'IPython.utils.encoding',
'IPython.utils.frame',
'IPython.utils.generics',
'IPython.utils.importstring',
'IPython.utils.io',
'IPython.utils.ipstruct',
'IPython.utils.module_paths',
'IPython.utils.openpy',
'IPython.utils.path',
'IPython.utils.process',
'IPython.utils.py3compat',
'IPython.utils.sentinel',
'IPython.utils.strdispatch',
'IPython.utils.sysinfo',
'IPython.utils.syspathcontext',
'IPython.utils.terminal',
'IPython.utils.text',
'IPython.utils.timing',
'IPython.utils.wildcard',
'__mp_main__',
'_asyncio',
'_contextvars',
'_cython_0_29_22',
'_cython_0_29_23',
'_elementtree',
'_lsprof',
'_multiprocessing',
'_sqlite3',
'_ssl',
'_sysconfigdata__linux_x86_64-linux-gnu',
'argparse',
'asttokens',
'asttokens.asttokens',
'asttokens.line_numbers',
'asttokens.mark_tokens',
'asttokens.util',
'asyncio',
'asyncio.base_events',
'asyncio.base_futures',
'asyncio.base_subprocess',
'asyncio.base_tasks',
'asyncio.constants',
'asyncio.coroutines',
'asyncio.events',
'asyncio.exceptions',
'asyncio.format_helpers',
'asyncio.futures',
'asyncio.locks',
'asyncio.log',
'asyncio.protocols',
'asyncio.queues',
'asyncio.runners',
'asyncio.selector_events',
'asyncio.sslproto',
'asyncio.staggered',
'asyncio.streams',
'asyncio.subprocess',
'asyncio.tasks',
'asyncio.threads',
'asyncio.transports',
'asyncio.trsock',
'asyncio.unix_events',
'backcall',
'backcall.backcall',
'bdb',
'cProfile',
'cmd',
'code',
'codeop',
'colorsys',
'concurrent',
'concurrent.futures',
'concurrent.futures._base',
'concurrent.futures.process',
'concurrent.futures.thread',
'contextvars',
'dataclasses',
'decorator',
'difflib',
'email._encoded_words',
'email._parseaddr',
'email._policybase',
'email.base64mime',
'email.charset',
'email.encoders',
'email.errors',
'email.feedparser',
'email.header',
'email.iterators',
'email.message',
'email.parser',
'email.quoprimime',
'email.utils',
'executing',
'executing.executing',
'executing.version',
'filecmp',
'fractions',
'getopt',
'gettext',
'html',
'html.entities',
'http',
'http.client',
'ipython_genutils',
'ipython_genutils._version',
'ipython_genutils.encoding',
'ipython_genutils.path',
'ipython_genutils.py3compat',
'ipython_genutils.text',
'jedi',
'jedi._compatibility',
'jedi.api',
'jedi.api.classes',
'jedi.api.completion',
'jedi.api.completion_cache',
'jedi.api.environment',
'jedi.api.errors',
'jedi.api.exceptions',
'jedi.api.file_name',
'jedi.api.helpers',
'jedi.api.interpreter',
'jedi.api.keywords',
'jedi.api.project',
'jedi.api.refactoring',
'jedi.api.refactoring.extract',
'jedi.api.strings',
'jedi.cache',
'jedi.common',
'jedi.debug',
'jedi.file_io',
'jedi.inference',
'jedi.inference.analysis',
'jedi.inference.arguments',
'jedi.inference.base_value',
'jedi.inference.cache',
'jedi.inference.compiled',
'jedi.inference.compiled.access',
'jedi.inference.compiled.getattr_static',
'jedi.inference.compiled.mixed',
'jedi.inference.compiled.subprocess',
'jedi.inference.compiled.subprocess.functions',
'jedi.inference.compiled.value',
'jedi.inference.context',
'jedi.inference.docstrings',
'jedi.inference.filters',
'jedi.inference.flow_analysis',
'jedi.inference.gradual',
'jedi.inference.gradual.annotation',
'jedi.inference.gradual.base',
'jedi.inference.gradual.conversion',
'jedi.inference.gradual.generics',
'jedi.inference.gradual.stub_value',
'jedi.inference.gradual.type_var',
'jedi.inference.gradual.typeshed',
'jedi.inference.gradual.typing',
'jedi.inference.gradual.utils',
'jedi.inference.helpers',
'jedi.inference.imports',
'jedi.inference.lazy_value',
'jedi.inference.names',
'jedi.inference.param',
'jedi.inference.parser_cache',
'jedi.inference.recursion',
'jedi.inference.references',
'jedi.inference.signature',
'jedi.inference.syntax_tree',
'jedi.inference.sys_path',
'jedi.inference.utils',
'jedi.inference.value',
'jedi.inference.value.decorator',
'jedi.inference.value.dynamic_arrays',
'jedi.inference.value.function',
'jedi.inference.value.instance',
'jedi.inference.value.iterable',
'jedi.inference.value.klass',
'jedi.inference.value.module',
'jedi.parser_utils',
'jedi.plugins',
'jedi.plugins.django',
'jedi.plugins.flask',
'jedi.plugins.pytest',
'jedi.plugins.registry',
'jedi.plugins.stdlib',
'jedi.settings',
'lxml',
'lxml._elementpath',
'lxml.etree',
'mimetypes',
'multiprocessing',
'multiprocessing.connection',
'multiprocessing.context',
'multiprocessing.pool',
'multiprocessing.process',
'multiprocessing.queues',
'multiprocessing.reduction',
'multiprocessing.util',
'networkx',
'networkx.algorithms',
'networkx.algorithms.assortativity',
'networkx.algorithms.assortativity.connectivity',
'networkx.algorithms.assortativity.correlation',
'networkx.algorithms.assortativity.mixing',
'networkx.algorithms.assortativity.neighbor_degree',
'networkx.algorithms.assortativity.pairs',
'networkx.algorithms.asteroidal',
'networkx.algorithms.bipartite',
'networkx.algorithms.bipartite.basic',
'networkx.algorithms.bipartite.centrality',
'networkx.algorithms.bipartite.cluster',
'networkx.algorithms.bipartite.covering',
'networkx.algorithms.bipartite.edgelist',
'networkx.algorithms.bipartite.generators',
'networkx.algorithms.bipartite.matching',
'networkx.algorithms.bipartite.matrix',
'networkx.algorithms.bipartite.projection',
'networkx.algorithms.bipartite.redundancy',
'networkx.algorithms.bipartite.spectral',
'networkx.algorithms.boundary',
'networkx.algorithms.bridges',
'networkx.algorithms.centrality',
'networkx.algorithms.centrality.betweenness',
'networkx.algorithms.centrality.betweenness_subset',
'networkx.algorithms.centrality.closeness',
'networkx.algorithms.centrality.current_flow_betweenness',
'networkx.algorithms.centrality.current_flow_betweenness_subset',
'networkx.algorithms.centrality.current_flow_closeness',
'networkx.algorithms.centrality.degree_alg',
'networkx.algorithms.centrality.dispersion',
'networkx.algorithms.centrality.eigenvector',
'networkx.algorithms.centrality.flow_matrix',
'networkx.algorithms.centrality.group',
'networkx.algorithms.centrality.harmonic',
'networkx.algorithms.centrality.katz',
'networkx.algorithms.centrality.load',
'networkx.algorithms.centrality.percolation',
'networkx.algorithms.centrality.reaching',
'networkx.algorithms.centrality.second_order',
'networkx.algorithms.centrality.subgraph_alg',
'networkx.algorithms.centrality.trophic',
'networkx.algorithms.centrality.voterank_alg',
'networkx.algorithms.chains',
'networkx.algorithms.chordal',
'networkx.algorithms.clique',
'networkx.algorithms.cluster',
'networkx.algorithms.coloring',
'networkx.algorithms.coloring.equitable_coloring',
'networkx.algorithms.coloring.greedy_coloring',
'networkx.algorithms.coloring.greedy_coloring_with_interchange',
'networkx.algorithms.communicability_alg',
'networkx.algorithms.community',
'networkx.algorithms.community.asyn_fluid',
'networkx.algorithms.community.centrality',
'networkx.algorithms.community.community_utils',
'networkx.algorithms.community.kclique',
'networkx.algorithms.community.kernighan_lin',
'networkx.algorithms.community.label_propagation',
'networkx.algorithms.community.lukes',
'networkx.algorithms.community.modularity_max',
'networkx.algorithms.community.quality',
'networkx.algorithms.components',
'networkx.algorithms.components.attracting',
'networkx.algorithms.components.biconnected',
'networkx.algorithms.components.connected',
'networkx.algorithms.components.semiconnected',
'networkx.algorithms.components.strongly_connected',
'networkx.algorithms.components.weakly_connected',
'networkx.algorithms.connectivity',
'networkx.algorithms.connectivity.connectivity',
'networkx.algorithms.connectivity.cuts',
'networkx.algorithms.connectivity.disjoint_paths',
'networkx.algorithms.connectivity.edge_augmentation',
'networkx.algorithms.connectivity.edge_kcomponents',
'networkx.algorithms.connectivity.kcomponents',
'networkx.algorithms.connectivity.kcutsets',
'networkx.algorithms.connectivity.stoerwagner',
'networkx.algorithms.connectivity.utils',
'networkx.algorithms.core',
'networkx.algorithms.covering',
'networkx.algorithms.cuts',
'networkx.algorithms.cycles',
'networkx.algorithms.d_separation',
'networkx.algorithms.dag',
'networkx.algorithms.distance_measures',
'networkx.algorithms.distance_regular',
'networkx.algorithms.dominance',
'networkx.algorithms.dominating',
'networkx.algorithms.efficiency_measures',
'networkx.algorithms.euler',
'networkx.algorithms.flow',
'networkx.algorithms.flow.boykovkolmogorov',
'networkx.algorithms.flow.capacityscaling',
'networkx.algorithms.flow.dinitz_alg',
'networkx.algorithms.flow.edmondskarp',
'networkx.algorithms.flow.gomory_hu',
'networkx.algorithms.flow.maxflow',
'networkx.algorithms.flow.mincost',
'networkx.algorithms.flow.networksimplex',
'networkx.algorithms.flow.preflowpush',
'networkx.algorithms.flow.shortestaugmentingpath',
'networkx.algorithms.flow.utils',
'networkx.algorithms.graph_hashing',
'networkx.algorithms.graphical',
'networkx.algorithms.hierarchy',
'networkx.algorithms.hybrid',
'networkx.algorithms.isolate',
'networkx.algorithms.isomorphism',
'networkx.algorithms.isomorphism.ismags',
'networkx.algorithms.isomorphism.isomorph',
'networkx.algorithms.isomorphism.isomorphvf2',
'networkx.algorithms.isomorphism.matchhelpers',
'networkx.algorithms.isomorphism.temporalisomorphvf2',
'networkx.algorithms.isomorphism.tree_isomorphism',
'networkx.algorithms.isomorphism.vf2userfunc',
'networkx.algorithms.link_analysis',
'networkx.algorithms.link_analysis.hits_alg',
'networkx.algorithms.link_analysis.pagerank_alg',
'networkx.algorithms.link_prediction',
'networkx.algorithms.lowest_common_ancestors',
'networkx.algorithms.matching',
'networkx.algorithms.minors',
'networkx.algorithms.mis',
'networkx.algorithms.moral',
'networkx.algorithms.node_classification',
'networkx.algorithms.node_classification.hmn',
'networkx.algorithms.node_classification.lgc',
'networkx.algorithms.node_classification.utils',
'networkx.algorithms.non_randomness',
'networkx.algorithms.operators',
'networkx.algorithms.operators.all',
'networkx.algorithms.operators.binary',
'networkx.algorithms.operators.product',
'networkx.algorithms.operators.unary',
'networkx.algorithms.planar_drawing',
'networkx.algorithms.planarity',
'networkx.algorithms.reciprocity',
'networkx.algorithms.regular',
'networkx.algorithms.richclub',
'networkx.algorithms.shortest_paths',
'networkx.algorithms.shortest_paths.astar',
'networkx.algorithms.shortest_paths.dense',
'networkx.algorithms.shortest_paths.generic',
'networkx.algorithms.shortest_paths.unweighted',
'networkx.algorithms.shortest_paths.weighted',
'networkx.algorithms.similarity',
'networkx.algorithms.simple_paths',
'networkx.algorithms.smallworld',
'networkx.algorithms.smetric',
'networkx.algorithms.sparsifiers',
'networkx.algorithms.structuralholes',
'networkx.algorithms.swap',
'networkx.algorithms.tournament',
'networkx.algorithms.traversal',
'networkx.algorithms.traversal.beamsearch',
'networkx.algorithms.traversal.breadth_first_search',
'networkx.algorithms.traversal.depth_first_search',
'networkx.algorithms.traversal.edgebfs',
'networkx.algorithms.traversal.edgedfs',
'networkx.algorithms.tree',
'networkx.algorithms.tree.branchings',
'networkx.algorithms.tree.coding',
'networkx.algorithms.tree.decomposition',
'networkx.algorithms.tree.mst',
'networkx.algorithms.tree.operations',
'networkx.algorithms.tree.recognition',
'networkx.algorithms.triads',
'networkx.algorithms.vitality',
'networkx.algorithms.voronoi',
'networkx.algorithms.wiener',
'networkx.classes',
'networkx.classes.coreviews',
'networkx.classes.digraph',
'networkx.classes.filters',
'networkx.classes.function',
'networkx.classes.graph',
'networkx.classes.graphviews',
'networkx.classes.multidigraph',
'networkx.classes.multigraph',
'networkx.classes.ordered',
'networkx.classes.reportviews',
'networkx.convert',
'networkx.convert_matrix',
'networkx.drawing',
'networkx.drawing.layout',
'networkx.drawing.nx_agraph',
'networkx.drawing.nx_pydot',
'networkx.drawing.nx_pylab',
'networkx.exception',
'networkx.generators',
'networkx.generators.atlas',
'networkx.generators.classic',
'networkx.generators.cographs',
'networkx.generators.community',
'networkx.generators.degree_seq',
'networkx.generators.directed',
'networkx.generators.duplication',
'networkx.generators.ego',
'networkx.generators.expanders',
'networkx.generators.geometric',
'networkx.generators.internet_as_graphs',
'networkx.generators.intersection',
'networkx.generators.interval_graph',
'networkx.generators.joint_degree_seq',
'networkx.generators.lattice',
'networkx.generators.line',
'networkx.generators.mycielski',
'networkx.generators.nonisomorphic_trees',
'networkx.generators.random_clustered',
'networkx.generators.random_graphs',
'networkx.generators.small',
'networkx.generators.social',
'networkx.generators.spectral_graph_forge',
'networkx.generators.stochastic',
'networkx.generators.sudoku',
'networkx.generators.trees',
'networkx.generators.triads',
'networkx.linalg',
'networkx.linalg.algebraicconnectivity',
'networkx.linalg.attrmatrix',
'networkx.linalg.bethehessianmatrix',
'networkx.linalg.graphmatrix',
'networkx.linalg.laplacianmatrix',
'networkx.linalg.modularitymatrix',
'networkx.linalg.spectrum',
'networkx.readwrite',
'networkx.readwrite.adjlist',
'networkx.readwrite.edgelist',
'networkx.readwrite.gexf',
'networkx.readwrite.gml',
'networkx.readwrite.gpickle',
'networkx.readwrite.graph6',
'networkx.readwrite.graphml',
'networkx.readwrite.json_graph',
'networkx.readwrite.json_graph.adjacency',
'networkx.readwrite.json_graph.cytoscape',
'networkx.readwrite.json_graph.jit',
'networkx.readwrite.json_graph.node_link',
'networkx.readwrite.json_graph.tree',
'networkx.readwrite.leda',
'networkx.readwrite.multiline_adjlist',
'networkx.readwrite.nx_shp',
'networkx.readwrite.nx_yaml',
'networkx.readwrite.pajek',
'networkx.readwrite.sparse6',
'networkx.relabel',
'networkx.release',
'networkx.testing',
'networkx.testing.test',
'networkx.testing.utils',
'networkx.utils',
'networkx.utils.contextmanagers',
'networkx.utils.decorators',
'networkx.utils.heaps',
'networkx.utils.mapped_queue',
'networkx.utils.misc',
'networkx.utils.random_sequence',
'networkx.utils.rcm',
'networkx.utils.union_find',
'nipype.pipeline.plugins',
'nipype.pipeline.plugins.base',
'nipype.pipeline.plugins.condor',
'nipype.pipeline.plugins.dagman',
'nipype.pipeline.plugins.debug',
'nipype.pipeline.plugins.ipython',
'nipype.pipeline.plugins.legacymultiproc',
'nipype.pipeline.plugins.linear',
'nipype.pipeline.plugins.lsf',
'nipype.pipeline.plugins.multiproc',
'nipype.pipeline.plugins.oar',
'nipype.pipeline.plugins.pbs',
'nipype.pipeline.plugins.pbsgraph',
'nipype.pipeline.plugins.semaphore_singleton',
'nipype.pipeline.plugins.sge',
'nipype.pipeline.plugins.sgegraph',
'nipype.pipeline.plugins.slurm',
'nipype.pipeline.plugins.slurmgraph',
'nipype.pipeline.plugins.somaflow',
'nipype.pipeline.plugins.tools',
'nipype.utils.profiler',
'parso',
'parso._compatibility',
'parso.cache',
'parso.file_io',
'parso.grammar',
'parso.normalizer',
'parso.parser',
'parso.pgen2',
'parso.pgen2.generator',
'parso.pgen2.grammar_parser',
'parso.python',
'parso.python.diff',
'parso.python.errors',
'parso.python.parser',
'parso.python.pep8',
'parso.python.prefix',
'parso.python.token',
'parso.python.tokenize',
'parso.python.tree',
'parso.tree',
'parso.utils',
'pdb',
'pexpect',
'pexpect.exceptions',
'pexpect.expect',
'pexpect.pty_spawn',
'pexpect.run',
'pexpect.spawnbase',
'pexpect.utils',
'pickleshare',
'pkgutil',
'pprint',
'profile',
'prompt_toolkit',
'prompt_toolkit.application',
'prompt_toolkit.application.application',
'prompt_toolkit.application.current',
'prompt_toolkit.application.dummy',
'prompt_toolkit.application.run_in_terminal',
'prompt_toolkit.auto_suggest',
'prompt_toolkit.buffer',
'prompt_toolkit.cache',
'prompt_toolkit.clipboard',
'prompt_toolkit.clipboard.base',
'prompt_toolkit.clipboard.in_memory',
'prompt_toolkit.completion',
'prompt_toolkit.completion.base',
'prompt_toolkit.completion.deduplicate',
'prompt_toolkit.completion.filesystem',
'prompt_toolkit.completion.fuzzy_completer',
'prompt_toolkit.completion.nested',
'prompt_toolkit.completion.word_completer',
'prompt_toolkit.data_structures',
'prompt_toolkit.document',
'prompt_toolkit.enums',
'prompt_toolkit.eventloop',
'prompt_toolkit.eventloop.async_generator',
'prompt_toolkit.eventloop.inputhook',
'prompt_toolkit.eventloop.utils',
'prompt_toolkit.filters',
'prompt_toolkit.filters.app',
'prompt_toolkit.filters.base',
'prompt_toolkit.filters.cli',
'prompt_toolkit.filters.utils',
'prompt_toolkit.formatted_text',
'prompt_toolkit.formatted_text.ansi',
'prompt_toolkit.formatted_text.base',
'prompt_toolkit.formatted_text.html',
'prompt_toolkit.formatted_text.pygments',
'prompt_toolkit.formatted_text.utils',
'prompt_toolkit.history',
'prompt_toolkit.input',
'prompt_toolkit.input.ansi_escape_sequences',
'prompt_toolkit.input.base',
'prompt_toolkit.input.defaults',
'prompt_toolkit.input.typeahead',
'prompt_toolkit.input.vt100_parser',
'prompt_toolkit.key_binding',
'prompt_toolkit.key_binding.bindings',
'prompt_toolkit.key_binding.bindings.auto_suggest',
'prompt_toolkit.key_binding.bindings.basic',
'prompt_toolkit.key_binding.bindings.completion',
'prompt_toolkit.key_binding.bindings.cpr',
'prompt_toolkit.key_binding.bindings.emacs',
'prompt_toolkit.key_binding.bindings.focus',
'prompt_toolkit.key_binding.bindings.mouse',
'prompt_toolkit.key_binding.bindings.named_commands',
'prompt_toolkit.key_binding.bindings.open_in_editor',
'prompt_toolkit.key_binding.bindings.page_navigation',
'prompt_toolkit.key_binding.bindings.scroll',
'prompt_toolkit.key_binding.bindings.vi',
'prompt_toolkit.key_binding.defaults',
'prompt_toolkit.key_binding.digraphs',
'prompt_toolkit.key_binding.emacs_state',
'prompt_toolkit.key_binding.key_bindings',
'prompt_toolkit.key_binding.key_processor',
'prompt_toolkit.key_binding.vi_state',
'prompt_toolkit.keys',
'prompt_toolkit.layout',
'prompt_toolkit.layout.containers',
'prompt_toolkit.layout.controls',
'prompt_toolkit.layout.dimension',
'prompt_toolkit.layout.dummy',
'prompt_toolkit.layout.layout',
'prompt_toolkit.layout.margins',
'prompt_toolkit.layout.menus',
'prompt_toolkit.layout.mouse_handlers',
'prompt_toolkit.layout.processors',
'prompt_toolkit.layout.screen',
'prompt_toolkit.layout.scrollable_pane',
'prompt_toolkit.layout.utils',
'prompt_toolkit.lexers',
'prompt_toolkit.lexers.base',
'prompt_toolkit.lexers.pygments',
'prompt_toolkit.mouse_events',
'prompt_toolkit.output',
'prompt_toolkit.output.base',
'prompt_toolkit.output.color_depth',
'prompt_toolkit.output.defaults',
'prompt_toolkit.output.vt100',
'prompt_toolkit.patch_stdout',
'prompt_toolkit.renderer',
'prompt_toolkit.search',
'prompt_toolkit.selection',
'prompt_toolkit.shortcuts',
'prompt_toolkit.shortcuts.dialogs',
'prompt_toolkit.shortcuts.progress_bar',
'prompt_toolkit.shortcuts.progress_bar.base',
'prompt_toolkit.shortcuts.progress_bar.formatters',
'prompt_toolkit.shortcuts.prompt',
'prompt_toolkit.shortcuts.utils',
'prompt_toolkit.styles',
'prompt_toolkit.styles.base',
'prompt_toolkit.styles.defaults',
'prompt_toolkit.styles.named_colors',
'prompt_toolkit.styles.pygments',
'prompt_toolkit.styles.style',
'prompt_toolkit.styles.style_transformation',
'prompt_toolkit.utils',
'prompt_toolkit.validation',
'prompt_toolkit.widgets',
'prompt_toolkit.widgets.base',
'prompt_toolkit.widgets.dialogs',
'prompt_toolkit.widgets.menus',
'prompt_toolkit.widgets.toolbars',
'pstats',
'pty',
'ptyprocess',
'ptyprocess.ptyprocess',
'ptyprocess.util',
'pure_eval',
'pure_eval.core',
'pure_eval.my_getattr_static',
'pure_eval.utils',
'pure_eval.version',
'pydoc',
'pydoc_data',
'pydoc_data.topics',
'pyexpat',
'pyexpat.errors',
'pyexpat.model',
'pygments',
'pygments.console',
'pygments.filter',
'pygments.filters',
'pygments.formatter',
'pygments.formatters',
'pygments.formatters._mapping',
'pygments.formatters.html',
'pygments.formatters.terminal256',
'pygments.lexer',
'pygments.lexers',
'pygments.lexers._mapping',
'pygments.lexers.python',
'pygments.modeline',
'pygments.plugin',
'pygments.regexopt',
'pygments.style',
'pygments.styles',
'pygments.token',
'pygments.unistring',
'pygments.util',
'quopri',
'runpy',
'scipy',
'scipy.__config__',
'scipy._distributor_init',
'scipy._lib',
'scipy._lib._ccallback',
'scipy._lib._ccallback_c',
'scipy._lib._pep440',
'scipy._lib._testutils',
'scipy._lib._threadsafety',
'scipy._lib._util',
'scipy._lib.decorator',
'scipy._lib.deprecation',
'scipy._lib.messagestream',
'scipy.constants',
'scipy.constants.codata',
'scipy.constants.constants',
'scipy.linalg',
'scipy.linalg._decomp_cossin',
'scipy.linalg._decomp_ldl',
'scipy.linalg._decomp_polar',
'scipy.linalg._decomp_qz',
'scipy.linalg._decomp_update',
'scipy.linalg._expm_frechet',
'scipy.linalg._fblas',
'scipy.linalg._flapack',
'scipy.linalg._flinalg',
'scipy.linalg._matfuncs_sqrtm',
'scipy.linalg._matfuncs_sqrtm_triu',
'scipy.linalg._procrustes',
'scipy.linalg._sketches',
'scipy.linalg._solve_toeplitz',
'scipy.linalg._solvers',
'scipy.linalg.basic',
'scipy.linalg.blas',
'scipy.linalg.cython_blas',
'scipy.linalg.cython_lapack',
'scipy.linalg.decomp',
'scipy.linalg.decomp_cholesky',
'scipy.linalg.decomp_lu',
'scipy.linalg.decomp_qr',
'scipy.linalg.decomp_schur',
'scipy.linalg.decomp_svd',
'scipy.linalg.flinalg',
'scipy.linalg.lapack',
'scipy.linalg.matfuncs',
'scipy.linalg.misc',
'scipy.linalg.special_matrices',
'scipy.sparse',
'scipy.sparse._csparsetools',
'scipy.sparse._index',
'scipy.sparse._matrix_io',
'scipy.sparse._sparsetools',
'scipy.sparse.base',
'scipy.sparse.bsr',
'scipy.sparse.compressed',
'scipy.sparse.construct',
'scipy.sparse.coo',
'scipy.sparse.csc',
'scipy.sparse.csgraph',
'scipy.sparse.csgraph._flow',
'scipy.sparse.csgraph._laplacian',
'scipy.sparse.csgraph._matching',
'scipy.sparse.csgraph._min_spanning_tree',
'scipy.sparse.csgraph._reordering',
'scipy.sparse.csgraph._shortest_path',
'scipy.sparse.csgraph._tools',
'scipy.sparse.csgraph._traversal',
'scipy.sparse.csgraph._validation',
'scipy.sparse.csr',
'scipy.sparse.data',
'scipy.sparse.dia',
'scipy.sparse.dok',
'scipy.sparse.extract',
'scipy.sparse.lil',
'scipy.sparse.linalg',
'scipy.sparse.linalg._expm_multiply',
'scipy.sparse.linalg._norm',
'scipy.sparse.linalg._onenormest',
'scipy.sparse.linalg.dsolve',
'scipy.sparse.linalg.dsolve._add_newdocs',
'scipy.sparse.linalg.dsolve._superlu',
'scipy.sparse.linalg.dsolve.linsolve',
'scipy.sparse.linalg.eigen',
'scipy.sparse.linalg.eigen.arpack',
'scipy.sparse.linalg.eigen.arpack._arpack',
'scipy.sparse.linalg.eigen.arpack.arpack',
'scipy.sparse.linalg.eigen.lobpcg',
'scipy.sparse.linalg.eigen.lobpcg.lobpcg',
'scipy.sparse.linalg.interface',
'scipy.sparse.linalg.isolve',
'scipy.sparse.linalg.isolve._gcrotmk',
'scipy.sparse.linalg.isolve._iterative',
'scipy.sparse.linalg.isolve.iterative',
'scipy.sparse.linalg.isolve.lgmres',
'scipy.sparse.linalg.isolve.lsmr',
'scipy.sparse.linalg.isolve.lsqr',
'scipy.sparse.linalg.isolve.minres',
'scipy.sparse.linalg.isolve.utils',
'scipy.sparse.linalg.matfuncs',
'scipy.sparse.sputils',
'scipy.spatial',
'scipy.spatial._distance_wrap',
'scipy.spatial._geometric_slerp',
'scipy.spatial._hausdorff',
'scipy.spatial._plotutils',
'scipy.spatial._procrustes',
'scipy.spatial._spherical_voronoi',
'scipy.spatial._voronoi',
'scipy.spatial.ckdtree',
'scipy.spatial.distance',
'scipy.spatial.kdtree',
'scipy.spatial.qhull',
'scipy.spatial.transform',
'scipy.spatial.transform._rotation_groups',
'scipy.spatial.transform._rotation_spline',
'scipy.spatial.transform.rotation',
'scipy.special',
'scipy.special._basic',
'scipy.special._comb',
'scipy.special._ellip_harm',
'scipy.special._ellip_harm_2',
'scipy.special._lambertw',
'scipy.special._logsumexp',
'scipy.special._spherical_bessel',
'scipy.special._ufuncs',
'scipy.special._ufuncs_cxx',
'scipy.special.orthogonal',
'scipy.special.sf_error',
'scipy.special.specfun',
'scipy.special.spfun_stats',
'scipy.version',
'sqlite3',
'sqlite3.dbapi2',
'ssl',
'stack_data',
'stack_data.core',
'stack_data.formatting',
'stack_data.utils',
'stack_data.version',
'sysconfig',
'timeit',
'traitlets',
'traitlets._version',
'traitlets.config',
'traitlets.config.application',
'traitlets.config.configurable',
'traitlets.config.loader',
'traitlets.traitlets',
'traitlets.utils',
'traitlets.utils.bunch',
'traitlets.utils.decorators',
'traitlets.utils.descriptions',
'traitlets.utils.getargspec',
'traitlets.utils.importstring',
'traitlets.utils.sentinel',
'tty',
'unicodedata',
'urllib.error',
'urllib.request',
'urllib.response',
'uu',
'wcwidth',
'wcwidth.table_wide',
'wcwidth.table_zero',
'wcwidth.unicode_versions',
'wcwidth.wcwidth',
'xml',
'xml.dom',
'xml.dom.NodeFilter',
'xml.dom.domreg',
'xml.dom.minicompat',
'xml.dom.minidom',
'xml.dom.xmlbuilder',
'xml.etree',
'xml.etree.ElementPath',
'xml.etree.ElementTree'}
That has very little to do with nipype.config or nipype.logging, both of which are non-optionally in nipype. The numbers reported by improf are VMS, so I'm not sure what you're seeing. I'm guessing the negative number (which shows reliably) indicates that we're calling gc.collect() somewhere during import.
Okay, yes, this is coming from importing numpy: https://github.com/numpy/numpy/issues/10455#issuecomment-360304921
Indeed, if I import numpy on standard settings of my computer, it adds almost all of those 4.7GB I'm seeing. When I did the same experiment after export OMP_NUM_THREADS=1 it just added 50MB.
cc/ @matthew-brett since we casually talked about this yesterday - he will be interested to learn about this.
The good news is that, besides an overdue revision of the import chains (i.e., there's no reason why importing config from nipype should load the full library in all its depth), we can effectively push the numpy import to the last minute, and completely avoid importing it in workflow creation.
Ah - I thought this had been fixed in more recent Numpy / OpenBLAS. What version of Numpy do you have?
But we can set export OMP_NUM_THREADS=1 for the mother process and set it to the adequate number elsewhere if needed.
I guess. Though this shouldn't be handled by nipype. It is reasonable for tools to respect the environment.
I guess I misread that you wanted to delay importing numpy until after the workflow was constructed, which doesn't seem possible.
I guess I misread that you wanted to delay importing numpy until after the workflow was constructed, which doesn't seem possible.
Correct, but in theory, jailing workflow instantiation as we currently are doing should contain the numpy import within the process. If you are not willing to jail, workflow creation should import numpy with OMP_NUM_THREADS=1. The performance hit is minimal because there aren't any costly operations at that point.
According to https://github.com/numpy/numpy/issues/21376, it seems that setting OPENBLAS_NUM_THREADS=1 will also work without interfering with the more commonly set OMP_NUM_THREADS. Confirmed with improf:
(base) ➜ ~ improf numpy numpy
numpy: 129 modules; 64.73ms, 313.1MiB
numpy: 0 modules; 0.09ms, 0.0MiB
Final: 129 modules; 64.82ms, 313.1MiB
Modules loaded with numpy:
set()
(base) ➜ ~ OMP_NUM_THREADS=1 improf numpy numpy
numpy: 129 modules; 55.70ms, 57.1MiB
numpy: 0 modules; 0.09ms, 0.0MiB
Final: 129 modules; 55.79ms, 57.1MiB
Modules loaded with numpy:
set()
(base) ➜ ~ OPENBLAS_NUM_THREADS=1 improf numpy numpy
numpy: 129 modules; 57.23ms, 57.1MiB
numpy: 0 modules; 0.08ms, 0.0MiB
Final: 129 modules; 57.31ms, 57.1MiB
Modules loaded with numpy:
set()
Are you then using OpenBLAS or MKL?
python -c 'import numpy; numpy.show_config()'
Looks like OpenBLAS - should I consider MKL?
python -c 'import numpy; numpy.show_config()'
blas_info:
libraries = ['cblas', 'blas', 'cblas', 'blas']
library_dirs = ['/home/oesteban/.miniconda/lib']
include_dirs = ['/home/oesteban/.miniconda/include']
language = c
define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
libraries = ['cblas', 'blas', 'cblas', 'blas']
library_dirs = ['/home/oesteban/.miniconda/lib']
include_dirs = ['/home/oesteban/.miniconda/include']
language = c
lapack_info:
libraries = ['lapack', 'blas', 'lapack', 'blas']
library_dirs = ['/home/oesteban/.miniconda/lib']
language = f77
lapack_opt_info:
libraries = ['lapack', 'blas', 'lapack', 'blas', 'cblas', 'blas', 'cblas', 'blas']
library_dirs = ['/home/oesteban/.miniconda/lib']
language = c
define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
include_dirs = ['/home/oesteban/.miniconda/include']
Supported SIMD extensions in this NumPy install:
baseline = SSE,SSE2,SSE3
found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2,AVX512F,AVX512CD,AVX512_SKX,AVX512_CLX
not found = AVX512_KNL,AVX512_KNM,AVX512_CNL,AVX512_ICL