pylsl cannot be used within a python multiprocessing.Process

[timonmerk]

In the ReadMe it is written "On Linux one currently cannot call pylsl functions from a thread that is not the main thread."

I want to use LSL within a Python multiprocessing.Process for reading in data from a stream (and I am using a Windows machine Python 3.10.4). I get the following exception however ctypes objects containing pointers cannot be pickled.

So I assume multiprocessing for pylsl is not possible due to the C type implementation right?

[sappelhoff]

@timonmerk I think the maintained repo of pylsl is https://github.com/labstreaminglayer/liblsl-Python

@cboulay @chkothe would it perhaps be reasonable to make people aware of that in the README of this repo?

[cboulay]

Hi @timonmerk! Sorry I didn't see this because it's not a repo that I maintain. @sappelhoff is correct; that's where you want to go for pylsl support. Please open an issue there if you want to discuss. Or you can find me on the LabStreamingLayer Slack.

I can give a brief answer here: (1) liblsl has a fully functional C-API and is pretty easy to wrap in a custom wrapper (cffi, pybind11) if you like, though I don't now if that will help with multiprocessing; (2) I think you can restructure your application so all of the LSL setup and teardown happens in the sub-process and the only things that go through IPC are the resolver predicate (a string) and the returned data (ndarray).

[matpadalino87]

Hi!!! @timonmerk have you solved your issue about lsl streaming with multiprocess Python module? I hope yes. I'm working on Spyder Conda and I've the same issue with multiprocess ( not multiprocessing )... I appreciated the comment of @cboulay but my level of coding with wrappers file is null! If I use threading module there are no problems with inlet function but I need to run 3/4 processes contemporary and so I'm actually stopped ... I add to this issue the description after the info about ctypes objects not be pickled for enhancing this discussion:

Traceback (most recent call last): File "", line 1, in File "C:\Users\Matteo\anaconda3\envs\ISO-CLS\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\Matteo\anaconda3\envs\ISO-CLS\lib\site-packages\multiprocess\spawn.py", line 126, in _main self = reduction.pickle.load(from_parent) File "C:\Users\Matteo\anaconda3\envs\ISO-CLS\lib\site-packages\dill_dill.py", line 373, in load return Unpickler(file, ignore=ignore, **kwds).load() File "C:\Users\Matteo\anaconda3\envs\ISO-CLS\lib\site-packages\dill_dill.py", line 646, in load obj = StockUnpickler.load(self) EOFError: Ran out of input

If there is someone that can help me, will have my esteem and attention!

[cboulay]

@matpadalino87 - This repo is not the official pylsl repo. Please go to https://github.com/labstreaminglayer/liblsl-Python and open an issue there or ind me on the LabStreamingLayer Slack.

@chkothe this particular issue isn't really a bug/problem with pylsl itself so much as it is a software architecture problem so I think this issue should be closed. That doesn't mean we have to stop the conversation, but for tracking purposes this should be closed.

[matpadalino87]

Dear @cboulay you are right! I switch now the conversation as you tell me.