[v4] Bundled Coroutine

[Erwsaym]

trafficstars

Hello, I'm currently playing with coroutine with Lua and Sol and I'm actually a bit confused. Indeed, I'm trying to reproduce something like that:

Lua Code:

co1 = coroutine.create(
	function()
	        print("AA : Step 1")
	        coroutine.yield()
	        print("AA : Step 2")
	end
)
coroutine.resume(co1)

co2 = coroutine.create(
	function()
	        print("BB : Step 1")
	        coroutine.yield()
	        print("BB : Step 2")
	end
)
coroutine.resume(co2)

coroutine.resume(co1)

Output:

$ lua test.lua
AA : Step 1
BB : Step 1
AA : Step 2

So I wanted to be able to do the same thing using sol::coroutine, but it seems that the wrong coroutine is executed:

#include <sol/sol.hpp>
#include <iostream>

int main()
{
    sol::state lua;
    lua.open_libraries(sol::lib::base, sol::lib::package, sol::lib::coroutine);

    lua["print"] = [](const sol::lua_value &v){
        std::cout << v.as<std::string>() << std::endl;
    };
    lua["cyield"] = sol::yielding([](){
        std::cout << "YIELDING" << std::endl;
    });

    sol::coroutine co1 = lua.load(R"(
        print("AA : Step 1")
        cyield()
        print("AA : Step 2")
    )");
    co1();

    sol::coroutine co2 = lua.load(R"(
        print("BB : Step 1")
        cyield()
        print("BB : Step 2")
    )");
    co2();

    co1();

    return 0;
}

Output :

AA : Step 1
YIELDING
AA : Step 2
AA : Step 1
YIELDING

I read some things about using sol::thread but I don't understand the way I should use it, for me I don't want to multithread my code, so I don't know if I'm on the good way.

Thanks for your help !

[ThePhD]

So, the reason you need sol::thread is because each of those coroutines needs its own execution stack. I call sol::thread by that name because that's how the Lua API and documentation refers to such an entity: a "thread".

coroutine.create(...) is hella wrong by Lua here, because coroutine.create doesn't return you a coroutine: it returns you a thread (an execution stack) with an associated coroutine.

In reality, it's closer to sol::execution_stack. It is a context in which the coroutine runs, and each coroutine needs its own sol::thread to have independent variables that don't clash with one another and its own return types. That's why the examples (https://sol2.readthedocs.io/en/latest/api/coroutine.html#yield-main-thread) go through some shenanigans, to give you the right code:

#include <sol/sol.hpp>
#include <iostream>

int main()
{
    sol::state lua;
    lua.open_libraries(sol::lib::base, sol::lib::package, sol::lib::coroutine);

    lua["print"] = [](sol::object v){
        std::cout << v.as<std::string>() << std::endl;
    };
    lua["cyield"] = sol::yielding([](){
        std::cout << "YIELDING" << std::endl;
    });

    // notice the new threads!
    sol::thread thread1 = sol::thread::create(lua);
    sol::thread thread2 = sol::thread::create(lua);

    // notice we load it FROM the new "execution stack"
    // we need it to have thread1's stack perspective
    sol::coroutine co1 = thread1.state().load(R"(
        print("AA : Step 1")
        cyield()
        print("AA : Step 2")
    )");
    // call first coroutine here
    co1();

    // notice we load it FROM the new "execution stack"
    // we need it to have thread2's stack and perspective
    sol::coroutine co2 = thread2.state().load(R"(
        print("BB : Step 1")
        cyield()
        print("BB : Step 2")
    )");

    // run the other coroutine
    co2();
    co1();
    // tada! they run on
    // independent stacks

    return 0;
}

Note to self

Given the number of times this has happened, I'm beginning to think I should write a sol::bundled_coroutine and completely change the docs to have people use that, where it automatically creates a new stack to be paired with every coroutine. The current interface is one-to-one with low-level concepts, and honestly that's causing people more heartache than benefit right now...

[ThePhD]

(To be clear, there is no sol::execution_stack. It doesn't exist; it's just a better name than sol::thread. Still, I need a good name for something like coroutine_but_it_has_the_thread_too...)

[Erwsaym]

Thanks a lot for your reply, indeed this naming and the context itself is a bit confusing, maybe something named like coroutine_context could make it ?

[ThePhD]

Current name idea so far is sol::packed_coroutine.

This is also going to make me create a sol3 header. where I create aliases under a sol3:: namespace using good names for everything....

[ThePhD]

Will be part of v4.

[davidchisnall]

I am not sure if this is the right place to post but I am similarly confused by how coroutines are exposed and the docs are somewhat sparse. I would like to be able to allow a function implemented in C++ to yield if the operation would block so that other Lua threads can run. The sol::yielding wrapper doesn't appear to provide a mechanism for capturing the continuation that allows me to return a value and all of the documentation for the sol::thread and sol::coroutine machinery seems to be assuming that I am creating a thread in C++, rather than allowing Lua code to create coroutines and resuming them in response to external events.

[sokolas]

@davidchisnall could you find a way to do that? I'm trying to implement the same funtionality you're talking about but every discussion or example I find here seems like a dead end.

[davidchisnall]

I did, a year ago, but I'm not sure I can distill a useful example. I'll try to get around to putting my code somewhere public soon. Reading through my code now:

I wrote a simple cooperative scheduler that exposed some blocking calls, including the option to run functions in the background as coroutines. The model for things involving the scheduler is that coroutines yield to the scheduler, which then yields to another thread. The scheduler keeps a sol::thread (stack) and a sol::coroutine (callable function that returns when it yields, constructed from the sol::thread and a sol::function). When Lua calls a function that you've exposed with sol::yielding, you need to capture the current coroutine (scheduler state) and later invoke it with whatever the result is.

I do this by packaging the current thread state (thread and coroutine) in an event object that has a callback that, when populated, returns the result (a sol::variadic_results). When the event is ready, it's pushed into a list of ready events. The scheduler iterates over this and then invokes the coroutine that the event captured with the result of its callback.

This is only a couple of hundred lines of code, including the kqueue integration, but reading the docs was frustrating because they assumed that I knew a lot more about the internals of both Sol and the Lua VM than I did.

The main problem is that the function invoked by sol::yielding does not take the current thread and coroutine as arguments. I couldn't find a way of extracting them from Sol3, so I ended up having my scheduler track them. This should be somewhat redundant (on invoking sol::yielding, Sol must at least know the current thread, and probably the current coroutine, so should be able to just pass them into the lambda).

I also needed to create the main thread like this:

        ThreadState(sol::state &state, sol::string_view code)
        {
            thread    = sol::thread::create(state);
            coroutine = thread.state().load(code);
        }

Without that, I could yield from the main thread but not get back to it.

If I remember correctly, calling a sol::yielding function causes the call into Lua to return back to C++. You can then pick another coroutine to run.