Automatically yield when calling an AsyncFunction

[daurnimator]

• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/AsyncFunction

[daurnimator]

I see an open (#2) issue from 2017 that seems related but hasn't been resolved. This problem is proving a bit of a blocker to my project at the moment, so I'd be happy to contribute a PR to move this forward, but I'm not sure exactly what the strategy should be. Did you have any ideas how to resolve #2?

@libitx, you'll need to experiment with using .then with lua_pcallk: the continuation is the .then piece.

[libitx]

Thanks for the response @daurnimator (and apologies for the dupe issue).

I'm a little confused as to where I even need to be looking. For example if I've added an async function like this:

const foo = async function() {
  // some async stuff
}

interop.push(L, foo)
lua.lua_setglobal(L, to_luastring('foo'))

When I call foo() in lua, as far as I can work out this code is responsible for calling it:

https://github.com/fengari-lua/fengari-interop/blob/master/src/js.js#L778-L793

So it seems to me the function foo always lives in JavaScript world (if that makes sense) and the result is pushed on to the lua stack. In this case that result is a promise. So I'm not too sure where lua_pcallk would fit in to this. Am I even looking at this correctly or am I miles off?

Any pointers you can offer would be much appreciated. I'd also be willing to put a bounty on this issue if that would be of interest to you.

[lorenzos]

In my experience, you won't make async functions work with the interop library as it is right now. I've managed to make async calls from LUA only when using the "vanilla" fengari LUA API. I'll briefly describe my way, maybe it could help you.

1. Every LUA code I run is wrapped in a coroutine, so it can be paused:

luaCode = "coroutine.wrap(function() " + luaCode + ";\n end)()";

2. JS async functions are written (or wrapped) making use of LUA yield/resume API (see LUA reference):

const foo = async function() { 
    // some async stuff
};
const fooLua = function(L) {
    foo().then(function() {
        fengari.lua.lua_resume(L, null, 0);
    });
    fengari.lua.lua_yield(L, 0);
};

3. The prepared function is finally passed to the LUA environment not by using interop, but with:

fengari.lua.lua_pushcfunction(L, fooLua);
fengari.lua.lua_setglobal(L, fengari.to_luastring('foo'));

There's a little more than that to handle LUA errors, JS exceptions, rejected promises, etc, but that's the minimum to make it work.

[libitx]

Thanks @lorenzos thats helpful. I've tried to follow that approach but still banging my head against a wall here. Do you mind if I share some code to see if one of you can point me in the right direction?

const { lua, lauxlib, lualib, to_luastring } = require('fengari')

// Create VM state
const L = lauxlib.luaL_newstate()
lualib.luaL_openlibs(L)


// Create async extension
const hello = async function(word) {
  console.log('JS', 'hello called')
  return new Promise(resolve => {
    setTimeout(_ => resolve(`hello ${ word }`), 10)
  })
}

lua.lua_pushcfunction(L, function() {
  console.log('JS', 'calling hello')
  hello('world')
    .then(res => {
      console.log('JS', 'async resolved', res)
      lua.lua_pushstring(L, to_luastring(res))
      lua.lua_resume(L, null, 0)
      console.log('JS', 'resumed')
    })
  console.log('JS', 'yielding')
  lua.lua_yield(L, 0)
  console.log('JS', 'yielded')
})

lua.lua_setglobal(L, to_luastring('hello'))


// Lua code
const code = `
coroutine.wrap(function()
  print('Lua '..'starting')
  local res = hello()
  print('Lua '..res)
end)()
`

// Execute
status = lauxlib.luaL_dostring(L, to_luastring(code))
console.log('JS', 'exec status', status)
if (status === 0) {
  console.log('JS', 'ok')
} else {
  const err = lua.lua_tojsstring(L, -1)
  console.log('JS', 'errored', err)
}

I've littered quite a lot of debug statements in the above to help see where I'm going wrong. When I run the above then this is what gets spit back at me:

Lua starting
JS calling hello
JS hello called
JS yielding
JS exec status 2
JS errored attempt to yield from outside a coroutine
JS async resolved hello world
JS resumed

What I want to see is Lua print out Lua hello world but I'm hitting issues. The error above suggests I'm attempting to yield from outside of a coroutine despite wrapping everything as suggested. So I think I have something fundamentally wrong first. Once I've understood what I'm doing wrong here I also need to wrap me head around how I would pass arguments from Lua to the JS function (eg pass the value world and receive hello world from the async function) without using fengari-interop... but one step at a time 😅

[lorenzos]

The error above suggests I'm attempting to yield from outside of a coroutine despite wrapping everything as suggested. So I think I have something fundamentally wrong first.

I can't try on live code now, but I see a couple things in your code that needs to be fixed. First a note. Any JS code after lua_yield() won't run ever, so the console.log('JS', 'yielded') line is useless. Then:

• Your "cfunction" implementation must declare L as its first and only argument, like this: lua_pushcfunction(L, function(L) { ... }). When called, the current LUA state will be there, and you should work on that state, not the global one. This may be the reason why you can't yield now.

• If you want to return something from your yielded function, you have to push it on the stack (and you're doing it) and pass the number of arguments returned to the resume function like this: lua_resume(L, null, 1).

I also need to wrap me head around how I would pass arguments from Lua to the JS function

You can still use interop for that, if the arguments to pass are simple scalar values. You just need to read them from the stack inside your "cfunction" implementation:

coroutine.wrap(function()
  local res = hello("world", 123)
end)()

lua.lua_pushcfunction(L, function(L) {
  var arg1 = interop.tojs(L, 1) // world (string)
  var arg2 = interop.tojs(L, 2) // 123 (number)
  console.log('JS', 'calling hello with', arg1, arg2)
  hello(arg1, arg2)
    .then(res => {
      // ...

[libitx]

Thank you, thats super helpful. I got this working in my sample script now - with arguments. 👍 Just need to figure out how to do this in my project which is a slightly more complex beast... but I'm on the right track. Thanks

[libitx]

I have another question. Suppose my wrapped code returns some value. How would I get that out of the coroutine?

In the code below the async hello function works nicely now. But how would I access that return value?

// Lua code
const code = `
coroutine.wrap(function()
  print('Lua '..'starting')
  print('Lua '..hello('world'))
  print('Lua '..'ending')
  return 123
end)()
`

// Execute
status = lauxlib.luaL_dostring(L, to_luastring(code))
console.log('JS', 'exec status', status)
if (status === 0) {
  console.log('JS', 'ok')
} else {
  const err = lua.lua_tojsstring(L, -1)
  console.log('JS', 'error', err)
}

EDIT

I've kind of solved this by wrapping the Lua code in coroutine.create instead of wrap, then pulling the thread from he stack. I can then start the thread and use setTimeout to poll the thread until it's dead. Once dead I can pull the return value from the thread's stack.

This works but does feel somewhat kludgy so please do let me know if you think there's a better approach.

[lorenzos]

Why do you need to have a return statement exactly like that? Can't you set a global or pass a callback function to get a "return" value in JS?

Anyway, instead of a poller to check when the thread is dead, I'd use a callback. For example, here's how I currently do in my project (again, this is simplified):

const executeAsyncLua = (code) => {
	// ...create state L, load libs, etc...
	return new Promise((resolve, reject) => {
		let wrappedCode = "function _script_run() " + code + ";\n end;\n _script_run();\n _script_exit_trap()";
		fengari.interop.push(L, () => resolve()); // <-- Here's the thing!
		fengari.lua.lua_setglobal(L, "_script_exit_trap");
		wrappedCode = "coroutine.wrap(function() " + wrappedCode + ";\n end)()";
		// ...load code, check syntax errors, execute, catch errors and call reject() in case...
	}
}

const luaCode = '...';
executeAsyncLua(luaCode).then(() => {
    // Here the script has run and finished
});

Another idea, if you still want to use the "return" statement as you showed, is to combine this with some more code wrapping trick to grab the returned value more "elegantly". Maybe wrap the original code into a function that get called, and its return value placed into a global or given into a JS callback.

[libitx]

The Lua code I'm running is user-generated code - always functions that return some value. So I can't change the Lua code at all, other than perhaps wrapping it in a coroutine as we've been discussing. I could also tail the user-code with some kind of exit trap function as you suggest above. Thanks for the idea 👍

[daurnimator]

In the code below the async hello function works nicely now

I don't see your hello function?

[libitx]

Hi guys, I just wanted to update you on my progress re the above discussion. I have now achieved what I wanted, following the approach suggested by @lorenzos (wrapping the Lua code in a coroutine). This approach involves not using fengari-interop for this specific purpose, which caused me to look a bit deeper into what interop was doing and I realised it didn't quite fit my use case perfectly. So I've ended up removing the fengari-interop dependency and creating my own similar but stripped back module that does just what I need but nothing more.

This discussion has been very helpful for me (thank you) but I don't think has moved you any closer to resolving the actual topic of this issue. 😄 But thanks all the same.