Cython extension imports broken

[standage]

The khmer and kevlar docs are both failing to build due to an import error that only appears when khmer is make install'd. The following error was nabbed from ReadTheDocs.

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/khmer/envs/master/lib/python3.5/site-packages/sphinx/cmdline.py", line 295, in main
    opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/khmer/envs/master/lib/python3.5/site-packages/sphinx/application.py", line 163, in __init__
    confoverrides or {}, self.tags)
  File "/home/docs/checkouts/readthedocs.org/user_builds/khmer/envs/master/lib/python3.5/site-packages/sphinx/config.py", line 134, in __init__
    execfile_(filename, config)
  File "/home/docs/checkouts/readthedocs.org/user_builds/khmer/envs/master/lib/python3.5/site-packages/sphinx/util/pycompat.py", line 129, in execfile_
    exec_(code, _globals)
  File "conf.py", line 59, in <module>
    from khmer._version import get_versions
  File "/home/docs/checkouts/readthedocs.org/user_builds/khmer/envs/master/lib/python3.5/site-packages/khmer-2.1.1+147.gcb717f4-py3.5-linux-x86_64.egg/khmer/__init__.py", line 517, in <module>
    from khmer._oxli.assembly import (LinearAssembler, SimpleLabeledAssembler,
  File "/home/docs/checkouts/readthedocs.org/user_builds/khmer/envs/master/lib/python3.5/site-packages/khmer-2.1.1+147.gcb717f4-py3.5-linux-x86_64.egg/khmer/_oxli/__init__.py", line 1, in <module>
    from .assembly import LinearAssembler
ImportError: No module named 'khmer._oxli.assembly'

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/khmer/envs/master/lib/python3.5/site-packages/khmer-2.1.1+147.gcb717f4-py3.5-linux-x86_64.egg/khmer/_oxli/__init__.py", line 1, in <module>
    from .assembly import LinearAssembler
ImportError: No module named 'khmer._oxli.assembly'

I can reproduce on my laptop by creating/activating a fresh 2.7 virtualenv, running "make clean install" in the khmer root directory, launching the python interpreter, and then trying to import khmer.

[standage]

Some relevant lines from the Makefile.

$(EXTENSION_MODULE): $(CPPSOURCES) $(CYSOURCES)
	./setup.py build_ext --inplace

So the Cython extensions are compiled and bundled into the khmer shared library by setup.py. Any ideas why things aren't lining up?

[camillescott]

Each cython extension file is its own extension class / shared lib -- not bundled into the khmer shared library. This looks like it's caused by combination of cython not being installed plus not distributing the generated cpp files.

On Wed, Jun 14, 2017 at 9:41 AM, Daniel Standage [email protected] wrote:

Some relevant lines from the Makefile.

$(EXTENSION_MODULE): $(CPPSOURCES) $(CYSOURCES) ./setup.py build_ext --inplace

So the Cython extensions are compiled and bundled into the khmer shared library by setup.py. Any ideas why things aren't lining up?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dib-lab/khmer/issues/1733#issuecomment-308489157, or mute the thread https://github.com/notifications/unsubscribe-auth/ACwxrR9ei-nsK95cN4YbGsFoaex1KikXks5sEA0hgaJpZM4N6IMG .

-- Camille Scott

Graduate Group for Computer Science Lab for Data Intensive Biology University of California, Davis

[email protected]

[standage]

Bingo. Installing cython manually before running make install, I can import khmer with no problem.

[standage]

So...we could easily add Cython to doc/requirements.txt, but really that's already handled by make install-dependencies. Any ideas about how to get ReadTheDocs to run make install-dependencies, or should we just take the quick-n-dirty approach and duplicate the requirement in doc/requirements.txt?

[camillescott]

There should probably just be a second target for RTD that only compiles the docs

On Jun 14, 2017 10:08 AM, "Daniel Standage" [email protected] wrote:

So...we could easily add Cython to doc/requirements.txt, but really that's already handled by make install-dependencies. Any ideas about how to get ReadTheDocs to run make install-dependencies, or should we just take the quick-n-dirty approach and duplicate the requirement in doc/requirements.txt?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dib-lab/khmer/issues/1733#issuecomment-308496816, or mute the thread https://github.com/notifications/unsubscribe-auth/ACwxrZICCqwPwXz_8jHiQxA0EagFnzS9ks5sEBOXgaJpZM4N6IMG .

[standage]

After a bit more thought: shouldn't we be distinguishing between installation dependencies and development dependencies? The DEVPKGS variable and the make install-dependencies target is great for setting up a dev environment, but perhaps it's time to use install_requires for screed and cython?

[camillescott]

Yeah, that stuff should be in setup.py, not the Makefile (doesn't the Makefile currently mirror that anyway?)

On Jun 14, 2017 10:32 AM, "Daniel Standage" [email protected] wrote:

After a bit more thought: shouldn't we be distinguishing between installation dependencies and development dependencies? The DEVPKGS variable and the make install-dependencies target is great for setting up a dev environment, but perhaps it's time to use install_requires for screed and cython?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dib-lab/khmer/issues/1733#issuecomment-308503280, or mute the thread https://github.com/notifications/unsubscribe-auth/ACwxrXmpmtUI56z_Qog1IAVg2-J7SI8Nks5sEBk0gaJpZM4N6IMG .

[standage]

*facepalm*

Updated. See https://github.com/dib-lab/khmer/pull/1734#issuecomment-308520289.

[standage]

This issue is cropping up for me again. The kevlar docs build on ReadTheDocs is failing again, and the only relevant difference I can see is that they recently upgraded to the latest version of pip.

I've created a clean sandbox for troubleshooting, but the deeper I dig the more confused I get. I can reproduce the ImportError: No module named 'khmer._oxli.assembly' error, but only when installing khmer to a virtualenv environment and running a khmer script. I see no errors when running a script after installing khmer to a venv virtual environment. However, in both cases, I see the errorImportError: No module named 'khmer._khmer' when I try to import khmer from the Python interpreter.

Inconsistencies in the Python packaging ecosystem are certainly a factor, but khmer's build process is bloated and needs some fixing and trimming down. I'm willing to spend some time on this, but I worry that most of the core contributors have lost interest and may not be excited about the time and attention needed to clean this up.

[standage]

Just tried Conda: get ImportError: No module named 'khmer._oxli.assembly' when attempting to run a script and ImportError: No module named 'khmer._khmer' when trying to import from the Python interpreter.

[ctb]

On Fri, Dec 21, 2018 at 07:35:30AM -0800, Daniel Standage wrote:

Inconsistencies in the Python packaging ecosystem are certainly a factor, but khmer's build process is bloated and needs some fixing and trimming down. I'm willing to spend some time on this, but I worry that most of the core contributors have lost interest and may not be excited about the time and attention needed to clean this up.

I'm game. In theory.... :)

[luizirber]

I started this some time ago, but it doesn't work yet:

https://github.com/dib-lab/khmer/compare/build/scikit-build

Based on scikit-build https://scikit-build.readthedocs.io/en/latest/

And since it uses cmake it also helps on building liboxli in a way that is easier to add to other projects.

On December 21, 2018 3:35:30 PM UTC, Daniel Standage [email protected] wrote:

This issue is cropping up for me again. The kevlar docs build on ReadTheDocs is failing again, and the only relevant difference I can see is that they recently upgraded to the latest version of pip.

I've created a clean sandbox for troubleshooting, but the deeper I dig the more confused I get. I can reproduce the ImportError: No module named 'khmer._oxli.assembly' error, but only when installing khmer to a virtualenv environment and running a khmer script. I see no errors when running a script after installing khmer to a venv virtual environment. However, in both cases, I see the errorImportError: No module named 'khmer._khmer' when I try to import khmer from the Python interpreter.

Inconsistencies in the Python packaging ecosystem are certainly a factor, but khmer's build process is bloated and needs some fixing and trimming down. I'm willing to spend some time on this, but I worry that most of the core contributors have lost interest and may not be excited about the time and attention needed to clean this up.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/dib-lab/khmer/issues/1733#issuecomment-449419562

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[standage]

I'm game. In theory.... :)

w00t!

I started this some time ago...Based on scikit-build...

Cool. Would you recommend we pick up from here, or is there value in starting again fresh?

[luizirber]

There isn't much there... feel free to take over =)

On December 21, 2018 4:45:47 PM UTC, Daniel Standage [email protected] wrote:

I'm game. In theory.... :)

w00t!

I started this some time ago...Based on scikit-build...

Cool. Would you recommend we pick up from here, or is there value in starting again fresh?

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/dib-lab/khmer/issues/1733#issuecomment-449438323

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[standage]

I just lost a few hours of my life troubleshooting the most bizarre behavior. Turns out if you open the Python interpreter and try to import khmer from within the khmer root directory, You're Gonna Have a Bad Time. Ugghhh...

In other bad news, I've been unable to reproduce the ImportError: No module named 'khmer._oxli.assembly' error that's occurring in the ReadTheDocs build.

Having zero experience with CMake I'm in over my head with the scikit-build stuff. I started from scratch whittling down the setup.py file, but it looks a lot like what Luiz came up with in his last attempt.

[luizirber]

I just lost a few hours of my life troubleshooting the most bizarre behavior. Turns out if you open the Python interpreter and try to import khmer from within the khmer root directory, You're Gonna Have a Bad Time. Ugghhh...

yeah... that's a reason why many people suggest putting code inside a src/ dir: https://hynek.me/articles/testing-packaging/

Having zero experience with CMake I'm in over my head with the scikit-build stuff. I started from scratch whittling down the setup.py file, but it looks a lot like what Luiz came up with in his last attempt.

I think there are two steps in this:

• generating cmake configs for building a static liboxli (kind of what the Makefile is doing nowadays). This also turns liboxli/khmer into something that can be imported in other C++ codebases.
• Point scikit-build to use step 1 to build the Cython extension. Leave cmake decide where zlib and bzip2 are coming from (preferably system).