poretools
poretools copied to clipboard
arq5x
docker build fails
I git cloned today, resulting in commit 2552313547985c1c7ff52d4301e401f8bada3d13
docker build -t poretools .
resulted in
============================================================================
* The following required packages can not be built:
* freetype
error: Setup script exited with 1
The command '/bin/sh -c cd /tmp/poretools && python setup.py install' returned a non-zero code: 1
Indeed. Looking further up we can see that error is from pip installing matplotlib-2.0.0b1 dependency:
Step 10 : RUN cd /tmp/poretools && python setup.py install
--snip--
BUILDING MATPLOTLIB
matplotlib: yes [2.0.0b1]
python: yes [2.7.6 (default, Jun 22 2015, 17:58:13) [GCC
4.8.2]]
platform: yes [linux2]
REQUIRED DEPENDENCIES AND EXTENSIONS
--snip--
freetype: no [The C/C++ header for freetype2 (ft2build.h)
could not be found. You may need to install the
development package.]
ft2build.h is installed by the libfreetype6-dev Ubuntu package:
$ dpkg -S `find /usr/include/ -name ft2build.h`
libfreetype6-dev: /usr/include/freetype2/ft2build.h
In spite of installing freetype, matplotlib still doesn't detected it. This was reported upstream (see matplotlib/matplotlib#3029), and the workaround was requiring pkg-config for matplotlib. Installing pkg-config for matplotlib allows matplotlib to install successfully, but then the h5py cythonize build step fails:
warning: no previously-included files found matching 'h5py/defs.pyx'
warning: no previously-included files found matching 'h5py/defs.pxd'
warning: no previously-included files found matching 'h5py/config.pxi'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs_api/_build'
warning: no previously-included files matching '.DS_Store' found under directory '*'
Autodetected HDF5 1.8.11
********************************************************************************
Summary of the h5py configuration
Path to HDF5: None
HDF5 Version: '1.8.11'
MPI Enabled: False
Rebuild Required: False
********************************************************************************
Executing api_gen rebuild of defs
Executing cythonize()
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/defs.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/_errors.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/_objects.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/_proxy.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5fd.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5z.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5i.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5r.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/utils.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/_conv.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5t.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5s.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5p.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5d.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5a.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5f.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5g.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5l.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5o.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5ds.pyx because it changed.
Compiling /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/h5ac.pyx because it changed.
[ 1/22] Cythonizing /tmp/easy_install-CDtEiR/h5py-2.6.0/h5py/_conv.pyx
Traceback (most recent call last):
File "setup.py", line 41, in <module>
'Topic :: Scientific/Engineering :: Bio-Informatics'
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 73, in run
self.do_egg_install()
File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 96, in do_egg_install
cmd.run()
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 381, in run
self.easy_install(spec, not self.no_deps)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 597, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 648, in install_item
self.process_distribution(spec, dist, deps)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 694, in process_distribution
[requirement], self.local_index, self.easy_install
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in best_match
return self.obtain(req, installer) # try and download/install
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in obtain
return installer(requirement)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 616, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 646, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 834, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1040, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1025, in run_setup
run_setup(setup_script, args)
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 50, in run_setup
lambda: execfile(
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 100, in run
return func()
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 52, in <lambda>
{'__file__':setup_script, '__name__':'__main__'}
File "setup.py", line 149, in <module>
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/dist-packages/setuptools/command/bdist_egg.py", line 185, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "/usr/lib/python2.7/dist-packages/setuptools/command/bdist_egg.py", line 171, in call_command
self.run_command(cmdname)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/dist-packages/setuptools/command/install_lib.py", line 21, in run
self.build()
File "/usr/lib/python2.7/distutils/command/install_lib.py", line 111, in build
self.run_command('build_ext')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/easy_install-CDtEiR/h5py-2.6.0/setup_build.py", line 189, in run
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 912, in cythonize
cythonize_one(*args)
File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 1017, in cythonize_one
result = compile([pyx_file], options)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 684, in compile
return compile_multiple(source, options)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 662, in compile_multiple
result = run_pipeline(source, options, context=context)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 492, in run_pipeline
err, enddata = Pipeline.run_pipeline(pipeline, source)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Pipeline.py", line 365, in run_pipeline
data = phase(data)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Pipeline.py", line 53, in generate_pyx_code_stage
module_node.process_implementation(options, result)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/ModuleNode.py", line 131, in process_implementation
self.generate_c_code(env, options, result)
File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/ModuleNode.py", line 328, in generate_c_code
rootwriter = Code.CCodeWriter()
File "Cython/Compiler/Code.py", line 1533, in Cython.Compiler.Code.CCodeWriter.__init__ (/tmp/pip_build_root/cython/Cython/Compiler/Code.c
:39203)
File "/usr/local/lib/python2.7/dist-packages/Cython/StringIOTree.py", line 15, in __init__
stream = StringIO()
TypeError: 'NoneType' object is not callable
This build error doesn't seem to be in the hdf5 upstream issue tracker. Interestingly hdf5 will compile and install from pip; just not from setup.py. One can see this interactively by instantiating the last valid Docker image when setup.py failed:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> fdb0f51c3e35 10 minutes ago 1.014 GB
$ docker run -it fdb0f51c3e35 /bin/bash
root@e67058055d67:/# cd /tmp/poretools/
root@e67058055d67:/tmp/poretools# python setup.py install # confirm this fails
root@e67058055d67:/tmp/poretools# pip install h5py # this works!?
root@e67058055d67:/tmp/poretools# python setup.py install # now this works
So after fixing matplotlib compilation, h5py fails, but only when installing via setup.py. Probably need more verbose output from setuptools to understand why.
Ah, so it turns out running setup.py a second time will successfully compile h5py:
$ docker run -it fdb0f51c3e35 /bin/bash
root@e67058055d67:/# cd /tmp/poretools/
root@e67058055d67:/tmp/poretools# python setup.py install # this fails
root@e67058055d67:/tmp/poretools# python setup.py install # now this works
However pip install h5py succeeds the first time. So something is wrong with using setup.py:
$ docker run -it fdb0f51c3e35 /bin/bash
root@e67058055d67:/# cd /tmp/poretools/
root@2f7f2ddec06b:/tmp/poretools# pip install ipython
root@2f7f2ddec06b:/tmp/poretools# ipython
In [1]: %run /usr/lib/python2.7/dist-packages/pip/__init__.py install h5py # successfully installs h5py
$ docker run -it fdb0f51c3e35 /bin/bash
root@e67058055d67:/# cd /tmp/poretools/
root@2f7f2ddec06b:/tmp/poretools# pip install ipython
root@2f7f2ddec06b:/tmp/poretools# ipython
In [1]: %run setup.py install
In [2]: %debug
ipdb> StringIO()
*** TypeError: 'NoneType' object is not callable
ipdb> l 1
1 try:
2 from cStringIO import StringIO
3 except ImportError:
4 from io import StringIO # does not support writing 'str' in Py2
5
6
7 class StringIOTree(object):
8 """
9 See module docs.
10 """
11
ipdb> l 11
6
7 class StringIOTree(object):
8 """
9 See module docs.
10 """
11
12 def __init__(self, stream=None):
13 self.prepended_children = []
14 if stream is None:
---> 15 stream = StringIO()
16 self.stream = stream
ipdb> ?StringIO
*** No help on StringIO
ipdb> type(StringIO)
<type 'NoneType'>
ipdb> import cStringIO
ipdb> cStringIO.StringIO
<built-in function StringIO>
ipdb> import io
ipdb> io.StringIO
<type '_io.StringIO'>
A web search of "python stringiotree.py nonetype stringio" suggests this error can happen if the cython compiled extension modules were reloaded (see https://mail.python.org/pipermail/cython-devel/2014-January/003944.html). scipy is the only dependency (of pandas) that compiles any cython code.
The error is reproducible by editing requirements.txt to have just pandas and h5py (if remove everything except h5py, there is no error). Instead of using python setup.py install using pip install pandas h5py works without issue.
$ docker run -it fdb0f51c3e35 /bin/bash
root@e67058055d67:/# cd /tmp/poretools/
root@38a6b4a134d2:/tmp/poretools# cat requirements.txt
h5py>=2.2.0
matplotlib
seaborn
pandas
root@38a6b4a134d2:/tmp/poretools# sed -i '/pandas\|h5py/!d' requirements.txt
root@38a6b4a134d2:/tmp/poretools# cat requirements.txt
h5py>=2.2.0
pandas
root@2f7f2ddec06b:/tmp/poretools# python setup.py install # fails
$ docker run -it fdb0f51c3e35 /bin/bash
root@e67058055d67:/# cd /tmp/poretools/
root@6aaf22b67d1c:/tmp/poretools# pip install pandas 'h5py>=2.2.0' # works
Edit: added required quoting around h5py in pip install pandas 'h5py>=2.2.0'
The NoneType error is not unique to installing h5py or pandas; it appears when installing more than 1 package interacting with cython. The error persists with the latest versions of setuptools and cython. As there doesn't seem to be a trivial way address cython module reloading in setuptools, one workaround is to make pip responsible for installation of cython dependent packages instead of setuptools. Determining each package's cython dependence would be wordy, so one can simply install all requirements via pip.
Pull request incoming.
There's a numpy ABI runtime error with the Docker image:
$ docker run -i poretools
Traceback (most recent call last):
File "/usr/local/bin/poretools", line 9, in <module>
load_entry_point('poretools==.0.5.1', 'console_scripts', 'poretools')()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 351, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2363, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2088, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "/usr/local/lib/python2.7/dist-packages/poretools-.0.5.1-py2.7.egg/poretools/__init__.py", line 4, in <module>
from Fast5File import *
File "/usr/local/lib/python2.7/dist-packages/poretools-.0.5.1-py2.7.egg/poretools/Fast5File.py", line 6, in <module>
import h5py
File "/usr/local/lib/python2.7/dist-packages/h5py/__init__.py", line 34, in <module>
from ._conv import register_converters as _register_converters
File "h5t.pxd", line 14, in init h5py._conv (/tmp/pip_build_root/h5py/h5py/_conv.c:6880)
File "numpy.pxd", line 66, in init h5py.h5t (/tmp/pip_build_root/h5py/h5py/h5t.c:19113)
ValueError: numpy.dtype has the wrong size, try recompiling
The numpy 1.8.2 upgrade to 1.11.0 is triggered when installing poretools' dependencies in requirements.txt. Since pip builds all packages before attempting to install them, one would need to upgrade numpy before installing any numpy dependent packages to ensure they all use the same ABI. One needs to use easy_install instead of pip to take precedence over the system numpy (see http://askubuntu.com/a/732371).
Just dropped in to say that I encountered the same error when I attempted to build on Ubuntu 15 with commit beb22ad.
============================================================================
* The following required packages can not be built:
* freetype
error: Setup script exited with 1
The command '/bin/sh -c cd /tmp/poretools && python setup.py install' returned a non-zero code: 1
