netcdf4-python icon indicating copy to clipboard operation
netcdf4-python copied to clipboard

Published 20 hours ago • verified publisher icon Unidata

netcdf4-python debug installs

Open astrojuanlu opened this issue 5 years ago • 4 comments

I am trying to debug https://github.com/Unidata/netcdf4-python/issues/814 and for that I would like to cythonize the sources with debugging information as suggested in https://cython.readthedocs.io/en/latest/src/userguide/debugging.html#debugging-your-cython-program. However, since https://github.com/Unidata/netcdf4-python/commit/afaf0fa68ad96fda272a33cb00433681dd6245bd#diff-2eeaed663bd0d25b7e608891384b7298, cythonize is not used and I don't see where to pass the option. Trying to reintroduce it in a naïve way:

diff --git a/setup.py b/setup.py
index c2a76da..c0b3869 100644
--- a/setup.py
+++ b/setup.py
@@ -2,6 +2,7 @@ import os, sys, subprocess
 import os.path as osp
 from setuptools import setup, Extension
 from distutils.dist import Distribution
+from Cython.Build import cythonize
 
 setuptools_extra_kwargs = {
     "install_requires": ["numpy>=1.7","cftime"],
@@ -609,5 +610,5 @@ setup(name="netCDF4",
                    "Topic :: System :: Archiving :: Compression",
                    "Operating System :: OS Independent"],
       packages=['netCDF4'],
-      ext_modules=ext_modules,
+      ext_modules=cythonize(ext_modules, gdb_debug=True),
       **setuptools_extra_kwargs)

results in an error:

Traceback (most recent call last):
  File "setup.py", line 613, in <module>
    ext_modules=cythonize(ext_modules, gdb_debug=True),
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 966, in cythonize
    aliases=aliases)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 824, in create_extension_list
    kwds = deps.distutils_info(file, aliases, base).values
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 679, in distutils_info
    return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 689, in transitive_merge
    node, extract, merge, seen, {}, self.cimported_files)[0]
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 694, in transitive_merge_helper
    deps = extract(node)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 659, in distutils_info0
    cimports, externs, incdirs = self.cimports_externs_incdirs(filename)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Utils.py", line 54, in wrapper
    res = cache[args] = f(self, *args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 547, in cimports_externs_incdirs
    for include in self.included_files(filename):
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Utils.py", line 54, in wrapper
    res = cache[args] = f(self, *args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 529, in included_files
    include_path = self.context.find_include_file(include, None)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 285, in find_include_file
    error(pos, "'%s' not found" % filename)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Errors.py", line 178, in error
    raise InternalError(message)
Cython.Compiler.Errors.InternalError: Internal compiler error: 'constants.pyx' not found

astrojuanlu avatar Jun 30 '19 16:06 astrojuanlu

I saw that this was introduced in https://github.com/Unidata/netcdf4-python/pull/673, which links to "issue #666", but I guess it's in some old issue tracker because it's a different thing.

If setup_requires was problematic, I wonder if using pyproject.toml as per PEP 518 would be a good replacement:

https://www.python.org/dev/peps/pep-0518/#build-system-table

astrojuanlu avatar Jun 30 '19 16:06 astrojuanlu

Turns out I was one step away:

diff --git a/setup.py b/setup.py
index c2a76da..e2aec1f 100644
--- a/setup.py
+++ b/setup.py
@@ -2,6 +2,7 @@ import os, sys, subprocess
 import os.path as osp
 from setuptools import setup, Extension
 from distutils.dist import Distribution
+from Cython.Build import cythonize
 
 setuptools_extra_kwargs = {
     "install_requires": ["numpy>=1.7","cftime"],
@@ -609,5 +610,5 @@ setup(name="netCDF4",
                    "Topic :: System :: Archiving :: Compression",
                    "Operating System :: OS Independent"],
       packages=['netCDF4'],
-      ext_modules=ext_modules,
+      ext_modules=cythonize(ext_modules, include_path=['include'], gdb_debug=True),
       **setuptools_extra_kwargs)

astrojuanlu avatar Jun 30 '19 16:06 astrojuanlu

As a last note, for some reason I was getting a really long compilation error when the cftime dependency was being resolved (similar to https://github.com/cython/cython/issues/2730):

Searching for cftime
Reading https://pypi.org/simple/cftime/
Downloading https://files.pythonhosted.org/packages/7a/83/a61141ec141ceb0617468e04cc163dbdb9007b958191043618d1dc950b8f/cftime-1.0.3.4.tar.gz#sha256=dd74d0d470baf1c50e31335215793a5e78436903e34b4f151fa9ccbf3a6cc20c
Best match: cftime 1.0.3.4
Processing cftime-1.0.3.4.tar.gz
Writing /tmp/easy_install-jtfol7od/cftime-1.0.3.4/setup.cfg
Running cftime-1.0.3.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-jtfol7od/cftime-1.0.3.4/egg-dist-tmp-iuz36oic
/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py:752: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  elif isinstance(patterns, basestring) or not isinstance(patterns, collections.Iterable):
Compiling /tmp/easy_install-jtfol7od/cftime-1.0.3.4/cftime/_cftime.pyx because it changed.
[1/1] Cythonizing /tmp/easy_install-jtfol7od/cftime-1.0.3.4/cftime/_cftime.pyx
Traceback (most recent call last):
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2215, in visit_FuncDefNode
    node.body = node.body.analyse_expressions(node.local_scope)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in analyse_expressions
    for stat in self.stats]
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in <listcomp>
    for stat in self.stats]
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 6665, in analyse_expressions
    self.target = self.target.analyse_target_types(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 7480, in analyse_target_types
    coerced_unpacked_item = unpacked_item.coerce_to(arg.type, env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 952, in coerce_to
    src = PyTypeTestNode(src, dst_type, env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 12999, in __init__
    assert dst_type.is_extension_type or dst_type.is_builtin_type, "PyTypeTest on non extension type"
AssertionError: PyTypeTest on non extension type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-jtfol7od/cftime-1.0.3.4/setup.py", line 96, in <module>
    elif line.startswith('#define NC_HAS_PNETCDF'):
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1096, in cythonize
    cythonize_one(*args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1202, in cythonize_one
    result = compile_single(pyx_file, options, full_module_name=full_module_name)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 727, in compile_single
    return run_pipeline(source, options, full_module_name)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 515, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 355, in run_pipeline
    data = run(phase, data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 335, in run
    return phase(data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1597, in __call__
    return super_class.__call__(root)
  File "Cython/Compiler/Visitor.py", line 355, in Cython.Compiler.Visitor.EnvTransform.__call__
  File "Cython/Compiler/Visitor.py", line 302, in Cython.Compiler.Visitor.CythonTransform.__call__
  File "Cython/Compiler/Visitor.py", line 284, in Cython.Compiler.Visitor.VisitorTransform.__call__
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1608, in visit_ModuleNode
    self.visitchildren(node)
  File "Cython/Compiler/Visitor.py", line 247, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 251, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1903, in visit_DefNode
    node = self.visit_FuncDefNode(node)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1895, in visit_FuncDefNode
    node.body.analyse_declarations(lenv)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 431, in analyse_declarations
    stat.analyse_declarations(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 1329, in analyse_declarations
    base_type = self.base_type.analyse(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 1084, in analyse
    axes_specs = MemoryView.get_axes_specs(env, self.axes)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/MemoryView.py", line 533, in get_axes_specs
    cythonscope.load_cythonscope()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/CythonScope.py", line 128, in load_cythonscope
    whitelist=MemoryView.view_utility_whitelist)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/UtilityCode.py", line 208, in declare_in_scope
    tree = self.get_tree(entries_only=True, cython_scope=cython_scope)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/UtilityCode.py", line 185, in get_tree
    (err, tree) = Pipeline.run_pipeline(pipeline, tree, printtree=False)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 355, in run_pipeline
    data = run(phase, data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 335, in run
    return phase(data)
  File "Cython/Compiler/Visitor.py", line 302, in Cython.Compiler.Visitor.CythonTransform.__call__
  File "Cython/Compiler/Visitor.py", line 284, in Cython.Compiler.Visitor.VisitorTransform.__call__
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2210, in visit_ModuleNode
    self.visitchildren(node)
  File "Cython/Compiler/Visitor.py", line 247, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 251, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 188, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 148, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error
Cython.Compiler.Errors.CompilerCrash: 
Error compiling Cython file:
------------------------------------------------------------
...
                            __pyx_memoryview_thread_locks[__pyx_memoryview_thread_locks_used], __pyx_memoryview_thread_locks[i])
                    break
            else:
                PyThread_free_lock(self.lock)

    cdef char *get_item_pointer(memoryview self, object index) except NULL:
        ^
------------------------------------------------------------

View.MemoryView:389:9: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(View.MemoryView:5:0)
StatListNode.stats[18] = StatListNode(View.MemoryView:330:5)
StatListNode.stats[0] = CnameDecoratorNode(View.MemoryView:330:5,
    cname = '__pyx_memoryview')
CnameDecoratorNode.node = CClassDefNode(View.MemoryView:330:5,
    as_name = 'memoryview',
    class_name = 'memoryview',
    decorators = [...]/0,
    module_name = '',
    visibility = 'private')
CClassDefNode.body = StatListNode(View.MemoryView:332:4)
StatListNode.stats[2] = CFuncDefNode(View.MemoryView:389:9,
    args = [...]/2,
    is_c_class_method = 1,
    modifiers = [...]/0,
    visibility = 'private')

Compiler crash traceback from this point on:
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2215, in visit_FuncDefNode
    node.body = node.body.analyse_expressions(node.local_scope)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in analyse_expressions
    for stat in self.stats]
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in <listcomp>
    for stat in self.stats]
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 6665, in analyse_expressions
    self.target = self.target.analyse_target_types(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 7480, in analyse_target_types
    coerced_unpacked_item = unpacked_item.coerce_to(arg.type, env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 952, in coerce_to
    src = PyTypeTestNode(src, dst_type, env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 12999, in __init__
    assert dst_type.is_extension_type or dst_type.is_builtin_type, "PyTypeTest on non extension type"
AssertionError: PyTypeTest on non extension type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 613, in <module>
    **setuptools_extra_kwargs)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/home/juanlu/.pyenv/versions/3.7.3-debug/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/juanlu/.pyenv/versions/3.7.3-debug/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/juanlu/.pyenv/versions/3.7.3-debug/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.run()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 418, in run
    self.easy_install(spec, not self.no_deps)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 660, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 707, in install_item
    self.process_distribution(spec, dist, deps)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 752, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
    replace_conflicting=replace_conflicting
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
    return self.obtain(req, installer)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
    return installer(requirement)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 705, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
    run_setup(setup_script, args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
    raise
  File "/home/juanlu/.pyenv/versions/3.7.3-debug/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/home/juanlu/.pyenv/versions/3.7.3-debug/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/_vendor/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-jtfol7od/cftime-1.0.3.4/setup.py", line 96, in <module>
    elif line.startswith('#define NC_HAS_PNETCDF'):
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1096, in cythonize
    cythonize_one(*args)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1202, in cythonize_one
    result = compile_single(pyx_file, options, full_module_name=full_module_name)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 727, in compile_single
    return run_pipeline(source, options, full_module_name)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 515, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 355, in run_pipeline
    data = run(phase, data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 335, in run
    return phase(data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1597, in __call__
    return super_class.__call__(root)
  File "Cython/Compiler/Visitor.py", line 355, in Cython.Compiler.Visitor.EnvTransform.__call__
  File "Cython/Compiler/Visitor.py", line 302, in Cython.Compiler.Visitor.CythonTransform.__call__
  File "Cython/Compiler/Visitor.py", line 284, in Cython.Compiler.Visitor.VisitorTransform.__call__
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1608, in visit_ModuleNode
    self.visitchildren(node)
  File "Cython/Compiler/Visitor.py", line 247, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 251, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1903, in visit_DefNode
    node = self.visit_FuncDefNode(node)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1895, in visit_FuncDefNode
    node.body.analyse_declarations(lenv)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 431, in analyse_declarations
    stat.analyse_declarations(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 1329, in analyse_declarations
    base_type = self.base_type.analyse(env)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 1084, in analyse
    axes_specs = MemoryView.get_axes_specs(env, self.axes)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/MemoryView.py", line 533, in get_axes_specs
    cythonscope.load_cythonscope()
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/CythonScope.py", line 128, in load_cythonscope
    whitelist=MemoryView.view_utility_whitelist)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/UtilityCode.py", line 208, in declare_in_scope
    tree = self.get_tree(entries_only=True, cython_scope=cython_scope)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/UtilityCode.py", line 185, in get_tree
    (err, tree) = Pipeline.run_pipeline(pipeline, tree, printtree=False)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 355, in run_pipeline
    data = run(phase, data)
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 335, in run
    return phase(data)
  File "Cython/Compiler/Visitor.py", line 302, in Cython.Compiler.Visitor.CythonTransform.__call__
  File "Cython/Compiler/Visitor.py", line 284, in Cython.Compiler.Visitor.VisitorTransform.__call__
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2210, in visit_ModuleNode
    self.visitchildren(node)
  File "Cython/Compiler/Visitor.py", line 247, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 251, in Cython.Compiler.Visitor.VisitorTransform.visitchildren
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 221, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 312, in Cython.Compiler.Visitor.CythonTransform.visit_Node
  File "Cython/Compiler/Visitor.py", line 260, in Cython.Compiler.Visitor.VisitorTransform._process_children
  File "Cython/Compiler/Visitor.py", line 219, in Cython.Compiler.Visitor.TreeVisitor._visitchildren
  File "Cython/Compiler/Visitor.py", line 193, in Cython.Compiler.Visitor.TreeVisitor._visitchild
  File "Cython/Compiler/Visitor.py", line 188, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "Cython/Compiler/Visitor.py", line 148, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error
setuptools.sandbox.UnpickleableException: CompilerCrash((<StringSourceDescriptor:View.MemoryView>, 389, 9), 'AnalyseExpressionsTransform', 'Compiler crash in AnalyseExpressionsTransform\n\nModuleNode.body = StatListNode(View.MemoryView:5:0)\nStatListNode.stats[18] = StatListNode(View.MemoryView:330:5)\nStatListNode.stats[0] = CnameDecoratorNode(View.MemoryView:330:5,\n    cname = \'__pyx_memoryview\')\nCnameDecoratorNode.node = CClassDefNode(View.MemoryView:330:5,\n    as_name = \'memoryview\',\n    class_name = \'memoryview\',\n    decorators = [...]/0,\n    module_name = \'\',\n    visibility = \'private\')\nCClassDefNode.body = StatListNode(View.MemoryView:332:4)\nStatListNode.stats[2] = CFuncDefNode(View.MemoryView:389:9,\n    args = [...]/2,\n    is_c_class_method = 1,\n    modifiers = [...]/0,\n    visibility = \'private\')\n\nCompiler crash traceback from this point on:\n  File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2215, in visit_FuncDefNode\n    node.body = node.body.analyse_expressions(node.local_scope)\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in analyse_expressions\n    for stat in self.stats]\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 436, in <listcomp>\n    for stat in self.stats]\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/Nodes.py", line 6665, in analyse_expressions\n    self.target = self.target.analyse_target_types(env)\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 7480, in analyse_target_types\n    coerced_unpacked_item = unpacked_item.coerce_to(arg.type, env)\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 952, in coerce_to\n    src = PyTypeTestNode(src, dst_type, env)\n  File "/home/juanlu/.pyenv/versions/dap37-debug/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 12999, in __init__\n    assert dst_type.is_extension_type or dst_type.is_builtin_type, "PyTypeTest on non extension type"\nAssertionError: PyTypeTest on non extension type', AssertionError('PyTypeTest on non extension type'), <traceback object at 0x7fd872cd8d78>)

(perhaps because I'm using Python with debugging symbols enabled and I had to install numpy and cython from source?)

This was fixed by removing the setup_requires line and doing pip install cftime right before python setup.py install.

astrojuanlu avatar Jun 30 '19 17:06 astrojuanlu

@Juanlu001 this is the setup_requires sandbox vs cython clash most likely (also reported here: https://github.com/pypa/setuptools/issues/1784 ) which happens when your own package uses Cython and has a setup_requires, rather than install_requires, dependency on another Cython-using package. One way to work around this is to get cftime package maintainers to call cythonize as subprocess.check_output([sys.executale, "-m", "Cython.Build.cythonize", ...]) in their setup.py/build_ext rather than importing it directly. Another way is to use a pyproject.toml with build-system.build_requires instead of setup_requires, even though that could cause issues if you ever plan an Android build for other reasons (it's complicated :grimacing: )

ghost avatar Oct 07 '19 11:10 ghost