Distributing pylsl on Linux

[cboulay]

It looks like the huge effort from @tstenner and me to get a manylinux pip-installable pylsl was all for naught. While we succeeded in creating the manylinux-compatible wheel, it turns out that this is not in fact functional, at least on Ubuntu 18.04, because of some incompatibility between lslboost and pthread. See #5

Step 1: We need better testing on the build systems. I was only testing that the shared object could load and we could get the library version, but I didn't test anything that interacted with boost or threading, so that's how this problem was able to sneak through.

Step 2: We need an alternative way to distribute pylsl.

• Option A: We could go back to using a single wheel for all platforms, and that wheel will contain liblsl compiled for every possible platform. This will require weird naming schemes and a very large wheel even though individual platforms will only use 1/8th of the file size.

• Option B: pip and conda for win & mac, conda only for Linux.

• Option C: Only the Windows wheels have compiled binaries, on all other platforms...

  • rely on liblsl being installed on the system. (e.g.. /usr/local/lib); or
  • create a setup script that builds/downloads liblsl.

• Option D: Instead focus our efforts on fixing whatever the compatibility issue is between lslboost and (ancient?) pthread. This is probably a waste of time.

[tstenner]

Option A: We could go back to using a single wheel for all platforms, and that wheel will contain liblsl compiled for every possible platform. This will require weird naming schemes and a very large wheel even though individual platforms will only use 1/8th of the file size.

If there's no way to have one library for Linux, it won't stay at 8 libraries at for my taste the naming scheme is already worse than it should be.

Option B: pip and conda for win & mac, conda only for Linux.

Or B: pip and conda for win & mac, conda only for Linux until we figure out what's the problem.

Option C: Only the Windows wheels have compiled binaries, on all other platforms... rely on liblsl being installed on the system. (e.g.. /usr/local/lib

That'd be my first option. The CI already builds .deb files for ubuntu 16.04 and 18.04 and Fedora has RPMs. Maybe we could talk with the neurodebian guys so we'd have an official distribution channel. We'd need an official new release they could pick up but I think it's worth it.

I already had a setup script which we should be able to revive with minor alterations.

Option D: Instead focus our efforts on fixing whatever the compatibility issue is between lslboost and (ancient?) pthread. This is probably a waste of time.

I've got everything prepared to move from Boost.Thread to std::thread. If that doesn't help, there's still option C.

[tstenner]

Step 1: We need better testing on the build systems. I was only testing that the shared object could load and we could get the library version, but I didn't test anything that interacted with boost or threading, so that's how this problem was able to sneak through.

My (slightly longer) approach: python3 -c 'import pylsl; print(pylsl.library_info()); pylsl.stream_outlet(pylsl.stream_info());'

This caught several problems with pthreads, boost and mismatched kernel headers and is about as fast.

With the tktechdocker/manylinux:gcc8.3.0 container and a bit of compiler magic I was able to generate a shared library that compiled, passed auditwheel and was able to open an outlet on another PC afterwards, so hopefully this is solved by then.

The shared object file isn't specific to python, so I'd put the build script in the regular liblsl repository and just download it from here, so we can really test if it loads in the default manylinux container and put less strain on the ci for liblsl-Python.

[cboulay]

That's great, thanks!

Do you want to update the CI for liblsl or shall I? If me, then where can I find your dockerfile / build scripts?

[cboulay]

I just saw the AppVeyor fail logs, so I guess you're already on it. Let me know if there's anything I can do to help.

[tstenner]

And Docker on Appveyor produces a binary: https://ci.appveyor.com/api/buildjobs/7v79ltt6jq6vk29s/artifacts/build_manylinux%2Finstall%2Flib%2Fliblsl.so.1.13.0

Could you check if pypi accepts it?

[cboulay]

If I build the wheel locally with python setup.py sdist bdist_wheel then the created wheel has the tag linux_x86_64 and not manylinux1, so pypi doesn't accept it.

I would have to build the wheel on Appveyor, probably using cibuildwheel. Do you have an appveyor.yml that can build or fetch the so?

[tstenner]

If I build the wheel locally with python setup.py sdist bdist_wheel then the created wheel has the tag linux_x86_64 and not manylinux1, so pypi doesn't accept it.

Also after auditwheel repair?

[cboulay]

Right, I forgot about that step, sorry! auditwheel works, and uploading seems to have accepted it if not for the fact that there was once already a file with that name on pypi. I'd have to give it a new tag and full release if I wanted to upload, and for that I'd rather use the CI system and do everything at once.

[cboulay]

@tstenner If this is resolved then please close this issue.

[xloem]

I spent a little time checking on this issue. TLDR: It's a boost/glibc-version library incompatibility that no longer triggered by v1.13.6 . This group of errors should be fully resolved by moving from Boost.Thread to std::thread , although the question of ensuring that the built boost matches the configuration of the system's glibc would still remain.

I reproduced the package environment of early 2019 / late 2018 in an ubuntu 18.04 docker image, and was able to see the original error from #5 in version 1.13.1 . The error is gone in version 1.13.6 . Versions 1.13.2 - 1.13.5 did not install an accompanying liblsl binary for me to check against.

My debug rebuild of the failing release is at https://ci.appveyor.com/api/buildjobs/4l1fvo2nw6s085n5/artifacts/wheelhouse%2Fpylsl-1.13.1-cp36-cp36m-manylinux1_x86_64.whl and can be installed by passing that url to pip3 install.

Here's a short script that reproduces the RuntimeError exception when run with that wheel in ubuntu 18.04. This was condensed from a failing run of the muse-lsl project.

from pylsl import StreamInfo, StreamOutlet
info = StreamInfo('Test', 'EEG', 0, 400, 'float32', 'test')
outlet = StreamOutlet(info, 12)

The error appears to relate to building lslboost using a different glibc. The old pthread code required condition variables have a realtime clock ( https://sources.debian.org/src/glibc/2.28-8/fbtl/old_pthread_cond_init.c/#L37 ) but the built lslboost has BOOST_THREAD_INTERNAL_CLOCK_IS_MONO set, resulting in invalid condition variable attributes. I found this using the debug symbols in the build.

The changes I made to liblsl to build my test wheel are at https://github.com/xloem/liblsl-Python/commit/0bc0798128402e9309fb85e10e7195f588e6a039 :

• remove windows builds
• use cibuildwheel < 2
• use revision 1.13.0-b5 of liblsl
• replace use of ninja with normal make, as ninja's flags have changed
• build and use Debug instead of Release libraries

To get a stack trace of the error I used gdb --args python3 -m pdb test.py and set a catchpoint on thrown exceptions.

[xloem]

I'm noting that lsl can use the system boost rather than the bundled boost, now: https://github.com/sccn/liblsl/blob/master/CMakeLists.txt#L28 . That would probably make manylinux reliable again.

[cboulay]

Since this issue first popped up, we also ran into issues with users of the manylinux distribution using pylsl 1.14.x on non-common linuxes. pip installed our pylsl manylinux because hey "it's manylinux", but it turns out their linux wasn't one of the many.

We've already switched over to Option C:

Option C: Only the Windows wheels have compiled binaries, on all other platforms...

• rely on liblsl being installed on the system. (e.g.. /usr/local/lib)

This degrades user experience slightly for those users on the most common and tested linuxes (i.e., Ubuntu 18.04 and 20.04), but it no longer shuts out other users. One important category of user that is no longer shut out is the raspberry pi user, who previously would get stuck with an ancient pylsl (the latest one that was "all source" - i.e., no platform specific wheels) that inevitably didn't work. Now at least they get the latest version with instructions saying liblsl not found, please install ...

We are about to release liblsl 1.16, then I'll do some testing and make a new pylsl release, and finally update the conda packages for liblsl and maybe pylsl (pylsl depends on liblsl for mac and linux, but not Windows).

After that's all done, I will try to remember to close this issue.

[xloem]

It would be good to link all the issues encountered from somewhere so future developers can work on easing the user experience in a stable manner if they desire. There’s a manylinux compatibility matrix down the readme at https://github.com/pypa/manylinux, and if something there isn’t true it could be simply a bug on their end.

When I bump into this I usually expect pip to build pylsl from source when I install it.

Anyway it sounds like this issue is closable or will be soon, which is great news.

[cboulay]

I usually expect pip to build pylsl from source when I install it.

How does that work? install_tools would have to trigger a git clone of liblsl followed by make, then copy liblsl.so into the folder and finally install the python package? Is that a common workflow in other packages? I could try it out but I'd like to see some examples.

We'd need to add a flag if the user wants to reuse system liblsl.

[xloem]

An example package that does this is pandas. pip3 install --install-option --no-use-wheel pandas will build the binary extension on the user's computer, when pip3 is run. The --no-use-wheel is because pandas also provides binary wheels.

The pandas approach means using the extension building functionality of setuptools and/or distutils, which are python code that can be hooked to add steps if needed, such as checking if liblsl is already installed. Some info at https://docs.python.org/3/extending/building.html#building-c-and-c-extensions-with-distutils . I've seen other projects derive new classes from the setuptools/distutils classes, to extend their behavior.

Regarding git clone, it looks like if the python package is distributed as an sdist package, the liblsl sources could optionally be bundled already into the distribution, or git could be run upon install in setup.py if the user does not have liblsl. Not sure what the existing norm is.

[xloem]

Here's an SO post on calling out to e.g. cmake within setup.py: https://stackoverflow.com/a/48015772/129550

[xloem]

I looked at this a little recently. I think I might have made some false comments somewhere in or on this topic by accident, not sure.

Anyway, it seems it could be normative to use scikit-build for this.

EDIT: I would include the sources as a git submodule myself. I looked at pandas/zeromq and it looked like they downloaded theirs from the web in setup.py .

[cboulay]

The last release (and imminent 1.16.1 release) use Option C -- packed in wheel for Windows but all other platforms need to install liblsl separately. While this is a bit tougher on some users, it's overall more consistent and enables some users to get pylsl working where before it would have provided unintelligible errors.