libpd compatibility

[alfonso73]

Hi, is there any binary for Win10 64bit? In theory is pd-lua embeddable in libpd? I would like to understand if would be possible to integrate pd-lua in Camomile ( https://github.com/pierreguillot/Camomile ). I guess that for this to be working pd-lua have to support multi instance. It would be a marvellous addition! thanks for this great project

[porres]

In theory is pd-lua embeddable in libpd?

yes, it has just been included in PlugData by the way

[agraef]

It should be noted, however, that nobody (except maybe Claude) really knows how reentrant and thread-safe pdlua.c interface is, and thus how the external behaves in a host which can run multiple plugdata plugins all at once and in different threads. That still needs to be investigated and, if needed, fixed.

[agraef]

FWIW, two plugdata instances in Bespoke with the same pd-lua object in a patch behave erratically, so I think that, yes, we'll have to work on the multi-instance/thread support.

UPDATE: This was a user error, not related to multi-threading or even pdlua. See my followup message below.

[agraef]

Yes, after looking at the source I can say that there are some issues which keep pd-lua from being truly multi-instance for now.

First of all, there's the global interpreter instance __L. We can't really get rid of this, thus we'll have to make it local to the plugin instance (if that is possible, I imagine that this will probably need some tie-in with plugdata to work), or at least thread-local (this requires static linkage, which is what we already do with plugdata anyway, so this should be easy to do).

There are three more required static variables, all related to proxy management, which will probably need to be given the same kind of treatment.

I'll keep this for the next release, so that we have a stable single-instance base to start work on this. But with some luck, 0.11.0 will hopefully be the last release that's not truly libpd-ready.

[porres]

can we change the title of this issue for something like 'multi instance support in libpb'? As I see it you offer binaries for windows.

[timothyschoen]

@agraef I think making it local to the plugin instance will be very difficult, if not impossible. TLS would be the best solution. It will still require us to fork lua I guess?

[agraef]

@timothyschoen Thanks for clarifying that, I was going to ask you about this. :) So TLS it is then.

As far as Lua itself is concerned, I think that no special adjustments will be needed there, we just need to make sure that the global Lua state __L in pdlua is thread-local. From the Lua wiki (http://lua-users.org/wiki/ThreadsTutorial):

Each thread in C which interacts with Lua will need its own Lua state. Each of these states has its own runtime stack.

[timothyschoen]

That's great news, I expected it to be way more difficult :)

[agraef]

Well, I have been dabbling with the TLS idea, but that just doesn't fly the way that pdlua is designed. Fortunately, as it turns out the current design with the single global interpreter is already thread-safe. That's because Lua itself (its C API, more precisely) already protects against concurrent uses. No TLS needed after all.

NB: The one instance I mentioned above, where I thought I saw erratic behaviour with pdlua in Bespoke wasn't actually a thread-related issue, and not even related to pdlua. All the other examples I tried with multiple plugdata plugins with pdlua objects in Bespoke worked just fine, no crashes at all, even with nasty stuff like reloading a pdlua class while the object keeps executing. If there really was an issue there, I would have seen crashes left and right by now.

Anyway, this is good news indeed, because it means that we can keep things simple. Of course, having what effectively amounts to a GIL (global interpreter lock) isn't a great design (ask the Python devs about it), but hey at least it works. ;-) And if you really need to work around this bottleneck then it's always possible to defer the heavy-duty processing to a Lua background thread.

So the current resolution is, yes, libpd works with pdlua just fine, so I'm keeping things as they are for now. However, I'm still keeping this issue open as a reminder that we might want to think about a truly multi-threaded design for pd-lua which does away with the GIL. Might be a nice project for a master thesis. :)

[agraef]

For the record, this is one of the examples of a Bespoke project that I've been using for testing purposes. It includes three plugdata instances, two of which use pdlua (these are just examples from the pdlua tutorial, slightly modified).

EDIT: The previous version didn't quite work right (Bespoke's karplusstrong swallowed most of the notes from the Fibonacci sequence, I'm not sure why), so I uploaded a new one which uses Vital instead. If you don't have that VST synth installed, you can use whatever you have, or connect the midioutput to an external Qsynth or the like. I also added some comments to make it easier to spot the plugdata instances; there are two examples from the pdlua tutorial in there (fibs and tictoc) and a simple additive synth with 4 voices which is just plain pd.

multi-pdlua-2023-01-18_16-13.bsk.zip

[agraef]

Fixed with #35 which also added multi-instance support to pd-lua.