Conda package for the python stfio module

[kh90909]

This issue is to document my experience building a Windows conda package for the python stfio module. While this does work, I find that it is relatively impractical at this point due to dependency issues. Should this change in future, I hope that the developers, or anyone else who is interested, can use my work as a starting point to make and distribute conda packages for stfio.

My motivation for building this package was so that I could have a single package file that I could easily install onto several Windows workstations, and be able to install the package into virtual environments for self-contained development.

For those that are not familiar with conda, it's the package manager used by the Anaconda python distribution. In contrast to pip, it allows packaging of non-python libraries, so it's more like apt-get or yum on Linux. This isn't a killer feature on Linux, since you can just use apt-get or yum, etc. to manage these non-python dependencies, but on Windows, it's very helpful since there is no such system package manager.

My work can be found in the conda branch of my fork. To build the package, you need the Anaconda python distribution, plus the same toolchain dependencies as Stimfit (VC++ 2008, Boost, SWIG and git) installed in the usual locations (see prerequisites.rst). You don't need the library dependencies (HDF5, Biosig); these are handled automatically. Then do:

git clone https://github.com/kh90909/stimfit.git
git checkout conda
cd dist/conda
conda build py-stfio

The build will finish with a message indicating the location of the .tar.bz2 conda package that you can install into your Anaconda installation, or a conda-managed virtual environment . You can also upload this package to the public binstar.org repository so that you can access it from other locations or share it with others. By default, conda builds for the architecture (i.e. 32 or 64-bit) of the machine the package is built on. To target a different architecture, try the instructions here. Conda is capable of building packages for Linux and Mac as well as Windows, however, my packages only contain Windows build scripts. They should work for both 32 and 64-bit architectures, but I have only tested them on 64-bit.

Sounds great, so far, but when you look under the hood to see what I had to do for the library dependencies, you'll see that updating these when new versions are released will be much less simple than it should be.

In an ideal world, suitable conda packages for hdf5* and biosig would be provided and maintained by their developers or the python distribution. In that case, everything would be simple. Back the real world, these packages don't exist, so I had to build them too. I had the choice of creating packages that build from source, or that grab the binaries from the appropriate website and package them up. I felt the former was untenable. With the burden of setting up and maintaining all the necessary build tools, the likelihood of anyone going to the trouble of building these packages is small. So, I built binary packages, and this is where things started to get complicated:

• The Windows VS2008 HDF5 binaries are only made available in self-installing exe form, which does not support a silent extract-only or install option. To get around this, I had to make a conda package for the command-line version of 7-zip which is needed to extract the full version of 7-zip which in turn is needed to extract the HDF5 exe!
• Recent Biosig Windows binary releases omit the DLLs, for no obvious reason. The most recent version that includes them is 1.5.6, so I had to make one change (kh90909/stimfit@cbd9ec9) to biosiglib.cpp in Stimfit to make it compatible.

*Anaconda does include a hdf5 package, but unfortunately it does not have the DLLs necessary for Stimfit/stfio. I'm discussing a fix with the Anaconda maintainers, so hopefully this will be resolved in a future version.

In conclusion, building these conda packages is possible, but it's somewhat impractical because the build recipes are not future proof due to the way the dependencies are distributed. I hope to get the issues with HDF5 resolved in the future, but I don't see the same happening with biosig unless someone steps up to build and maintain conda packages for it.

[neurodroid]

Thanks, this is very useful. I've merged your conda packaging directory into master and 0.14. A lot of scientific Python packages should depend on hdf5 (scipy, py-tables, py-hdf5, pandas etc.) - how are they packaged for conda?

[kh90909]

I took a look to see how other conda packages handle this...

1. pandas depends on scipy
2. scipy depends on h5py (I'm guessing this is what you mean by py-hdf5)
3. h5py and pytables bundle the hdf5 dlls

I couldn't find any info on how pytables is packaged or who does it, but h5py is packaged by the Anaconda maintainers. They pass the location of the hdf5 dlls/libs/headers to the distutils install script provided with h5py. The distutils script then includes those dlls in the package.

There's something a little strange here. According to the conda recipe, they appear to be depending on the conda hdf5 package to provide the dlls, but it does not! That's why I had to build my own version of the package. Either they use a different version of the hdf5 package internally, or they have manually installed hdf5 into their build system.

So, their approach was essentially the same as mine: depend on another package to provide the hdf5 dlls, headers, libs, and then inform the depending package where to find them.

[righeile]

I was trying to install stfio module as a conda package on mac, but did not succeed. It does install hdf5 and biosig, but does not build stfio. If anyone has any idea how to do it on mac, please let me know.

[neurodroid]

Can you try to use the new setup.py script to build stfio?

python setup.py build
python setup.py install

[righeile]

Thank you for the quick reply! I've tried it, it generated a bunch of warnings and failed in the end. Am I right that I can build it in any directory regardless where python is? Please find the error message attached, it is too long. errortext.txt

[neurodroid]

First, make sure that you use the correct Python version from the command line. For example, Anaconda Python is typically installed in $HOME/anaconda/bin/python. Second, you need the hdf5 development files. In a terminal, what's the output of mdfind -name libhdf5?

[kh90909]

@righeile, just to address the conda part of your question, the conda package will not build on Mac in its current state because there is only a Windows specific build recipe.

[righeile]

hdf5 are there. Thank you for your replies. I guess I will look for other solutions.

[neurodroid]

@righeile The easiest way to get it installed on OS X is to use MacPorts. After installation of MacPorts, run

sudo port install py27-stfio

[righeile]

I have installed it that way, but I cannot access stfio module in anaconda that way, just stimfit on its own.

[neurodroid]

@righeile If you have MacPorts installed, try running

CFLAGS="-I/opt/local/include" LDFLAGS="-L/opt/local/lib" python setup.py build
CFLAGS="-I/opt/local/include" LDFLAGS="-L/opt/local/lib" python setup.py install

Replace python with your Anaconda Python binary. Make sure you have all build dependencies (swig-python, boost, hdf5, libbiosig) installed via MacPorts.

[michal2am]

@kh90909 @neurodroid

Hi guys!

So I am trying to install stfio.

By package installation (Mint 64 bit):

sudo apt-get install python-stfio

and after launching python 2.7:

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import stfio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/stfio/__init__.py", line 8, in <module>
    from . import stfio_neo as neo
  File "/usr/lib/python2.7/dist-packages/stfio/stfio_neo.py", line 18, in <module>
    from neo.io.baseio import BaseIO
ImportError: No module named neo.io.baseio

However, I prefer to use it with Anaconda:

git clone https://github.com/kh90909/stimfit.git
git checkout conda
cd dist/conda
conda build py-stfio

and the result is:

Removing old build environment
Removing old work directory
BUILD START: py-stfio-trunk-py35_0
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ........
Solving package specifications: .
 Packages missing in current linux-64 channels: 
  - hdf5-dll
  - biosig
Missing dependency hdf5-dll, but found recipe directory, so building hdf5-dll first
 Packages missing in current linux-64 channels: 
  - hdf5-dll
  - biosig
Missing dependency biosig, but found recipe directory, so building biosig first
Error: The source field should be a dict, not str

So, is there any updated method to install stfio? I would prefer to have it in my Anaconda (Python 3.5.1 |Anaconda custom (64-bit)| ) but anything that works would be cool.

[neurodroid]

@michal2am I've just recently put together a setup.py script that you could try. First get all the build dependencies:

sudo apt-get build-dep stimfit

Get the stimfit source code:

git clone https://github.com/neurodroid/stimfit.git

Then build py-stfio:

cd stimfit
PYTHON setup.py build
PYTHON setup.py install

Replace PYTHON with the Python binary that you want to use. Let me know if this works for you. The error that you get with python-stfio from the Debian repository should be fixed by installing neo:

sudo apt-get install python-neo

But I'll also make sure that stfio can be loaded without neo in a future version.

[michal2am]

@neurodroid

Thanks for answer, so compilation and installation works on python 2, but importing the module, don't:

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import stfio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/stfio/__init__.py", line 6, in <module>
    from .stfio import *
  File "/usr/local/lib/python2.7/dist-packages/stfio/stfio.py", line 33, in <module>
    _stfio = swig_import_helper()
  File "/usr/local/lib/python2.7/dist-packages/stfio/stfio.py", line 25, in swig_import_helper
    import _stfio
ImportError: /usr/local/lib/python2.7/dist-packages/_stfio.so: undefined symbol: _ZN5stfio14importTDMSFileERKSsR9RecordingRNS_12ProgressInfoE

On python3, build is not working (full log attached). python3_build.txt

[neurodroid]

@michal2am The problem with python 2 should hopefully be fixed now:

cd stimfit
git pull origin master
python setup.py build
python setup.py install

Looks like you're missing the development files for python 3. Try

sudo apt-get install python3-dev

Let me know how this works

[michal2am]

@neurodroid

Thanks, now after fixes works in python2 and in fact I forgot to install dev for python 3 (and it works using python 3).

However, there is a problem with build with anaconda. If you are not familiar with Anaconda thats ok, but if you have free 5 minuts to look at log I will appreciate it:

python3_build_anaconda.txt

and hdf5 have been installed in my anaconda environment, also, adding this lapack (https://anaconda.org/cyclus/lapack) does not solve the problem.

[neurodroid]

@michal2am While I don't have anaconda installed, it would be great if you could help me fix this issue. A first step would be to edit setup.py. Could you comment out all keys that contain the string 'mkl' in the numpy_config_keys list and then run setup.py again. See https://github.com/neurodroid/stimfit/blob/master/setup.py#L12

[michal2am]

@neurodroid

sure, however tomorrow I am leaving for short holidays, so expect my answer within a week

[yueqiw]

Hello Christof, is there any update on this issue?

I figured out a way to install stfio for Anaconda Python on both Ubuntu 16.04 and macOS Sierra.

• I first tried python setup.py install. It didn't work. Then I tried make install:

Ubuntu 16.04

• I followed the instruction for Building only the standalone Python file i/o module
• At the configure step, I added --prefix to configure the installation path of stfio, so that it will be installed under the Anaconda site-packages folder

 $ ../../configure --enable-module --prefix=/Users/yueqi/anaconda/lib/python2.7/site-packages

• After make install, now stfio can be imported in Anaconda Python.

macOS Sierra

• I could not find a solution until I found this recent post: https://gehoon.github.io/note/Stimfit
• Basically, install all the missing dependencies by Homebrew. (the above post only listed boost and boost-python as dependencies, but in my case, I had to install several other packages as well).

$ brew update
$ brew install boost
$ brew install boost-python
$ brew install autoconf
$ brew install automake
$ brew install libtool
$ brew install fftw
$ brew install homebrew/science/hdf5

• Then follow the same build guide for Linux , but add --prefix to change the installation path to Anaconda.

$ git clone https://github.com/neurodroid/stimfit.git
$ cd stimfit
$ ./autogen.sh
$ mkdir build
$ mkdir build/module
$ cd build/module
$ ../../configure --enable-module --prefix=/Users/yueqi/anaconda/lib/python2.7/site-packages 
--with-hdf5-prefix=/usr/local/Cellar/hdf5/1.10.0-patch1/
$ make -j 4
$ make install

• The reason for adding -with-hdf5-prefix was because I encountered a syntax error in the configure file

   > ../../configure: line 17334: syntax error near unexpected token `HDF5,' 
   >../../configure: line 17334: `        PKG_CHECK_MODULES(HDF5, hdf5,'
  

  • To bypass this error, I had to delete the if ... ficode block surrounding this error line and explicitly specify the hdf5 path by-with-hdf5-prefix

• Now I can open a Jupyter notebook with all the conda packages and import stfio :)

[neurodroid]

On OS X I'm only actively supporting MacPorts as I don't have enough time to support any other Python distributions. However I'd appreciate if you could add your build instructions to the documentation. On any platform that does not come with a recent biosig library or no biosig library at all (to my knowledge that's every platform except of MacPorts, unless you've built biosig manually), I'd recommend to always configure --with-biosiglite to get full file i/o support.

[yueqiw]

Thanks for the fast reply.

I'm only working with Axon pClamp .abf file format, so stfio without biosig works for me already.

• I tried to install with configure --with-biosiglite. However, I got the following error:

>>> import stfio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/yueqi/anaconda/lib/python2.7/site-packages/stfio/__init__.py", line 6, in <module>
    from .stfio import *
  File "/Users/yueqi/anaconda/lib/python2.7/site-packages/stfio/stfio.py", line 23, in <module>
    _stfio = swig_import_helper()
  File "/Users/yueqi/anaconda/lib/python2.7/site-packages/stfio/stfio.py", line 22, in swig_import_helper
    return importlib.import_module('_stfio')
  File "/Users/yueqi/anaconda/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named _stfio

So I just went back to the version without configure --with-biosiglite

• It looks like Homebrew has support for latest version of biosig for MacOSX: http://biosig.sourceforge.net/download.html https://github.com/schloegl/homebrew-biosig I haven't tried it since I don't need biosig at this point.

I'll add the build instruction to the documentation. I'm not very familiar with GibHub, though. Should I just edit stimfit/doc/sphinx/osx_install_guide/building.rst ?

[neurodroid]

So I just went back to the version without configure --with-biosiglite

You'll have to completely remove your previous build (make clean) and your previous install (make uninstall) before configuring --with-biosiglite

Should I just edit stimfit/doc/sphinx/osx_install_guide/building.rst?

Yes please.

[kenshin87]

Is it possible to package the module to pypi so that we can easily install it with pip? With pip, it is much easier for us to use the package inside conda as a virtual env.

Thanks!

[kenshin87]

Is it possible to package the module to pypi so that we can easily install it with pip? With pip, it is much easier for us to use the package inside conda as a virtual env.

Thanks!

---

I tried to build with the guide from https://neurodroid.github.io/stimfit/linux_install_guide/moduleonly.html, but finally I failed.

Finally I gave up the idea of using anaconda, and just install the spyder IDE, since it uses the /usr/bin/python to run python commands, thus it works fine.

[yueqiw]

@kenshin87

I was able to build stfio for Anaconda on Unbuntu and MacOS. All you need to do is to direct the install path to where Anaconda is installed when building from source.

$ ../../configure --enable-module --prefix=$HOME/anaconda/

Check out this guide: https://github.com/yueqiw/stimfit/blob/conda-install/doc/sphinx/linux_install_guide/moduleonly.rst https://github.com/yueqiw/stimfit/blob/conda-install/doc/sphinx/osx_install_guide/moduleonly.rst

I did run into some kind of syntax error during the build, but it was resolved after deleting several lines in the configure file.

Hope this helps!

[kenshin87]

@yueqiw

Thanks a lot!

Last time since I failed to integrate stfio module into anaconda, considering the fact that conda has its own repo which is different from pypi, finally I switch to spyder and it works fine.