Suggestions for simplifying module creation

[alerque]

I'm having some trouble conceptualizing how to setup a project and am looking for any suggestions for how to keep it simple so I don't end up with 4 implementations of every function to maintain. I've had great success both using mlua as a runtime and for using it to create Lua modules. This projects needs to both at the same time and it's a bit of a mind bender.

The real project is SILE and my WIP is here, but I'll try to describe the relevant bits in MWE fashion here.

The first piece –the main project crate– is the CLI with a mlua based VM that loads a bunch of Lua code.

One complicating factor is that this needs to compile in several modes: with or without being built against system Lua sources (so the vendored feature), and also dynamic or static linking, and with the static option also a variant that has all the Lua resources (script and C modules) embedded in the binary. All that works already, but we have two more layers before things start getting weird.

I would also like to write code in Rust in the main Rust crate that is then exposed to in the end user in the Lua API. Again this is relatively easy to do directly ... some Rust functions and inject them into the Lua VM after instantiating it. In fact I started down just that read in this WIP. But it turns out this particular function was viable only because it could also be achieved on the Lua side with other dependencies. Because of the last requirement in this mess...

The last requirement is that the functions written in Rust and provided into Lua also need to be available if the Lua code is run from a system native Lua VM rather than the mlua based CLI wrapper. The whole thing needs to work as a Lua library module as well even without the CLI wrapper and mlua VM it provides.

In a naive attempt, I tried to simply add the module feature and export a Lua module from the existing crate lib.rs. That was a no-go for ugly linking reasons. Then I realized that was going to be a no-go anyway because of this limitation:

compile_error!("`vendored` and `module` features are mutually exclusive");

So my next move was to split it into 2 crates: one for the main app with the VM, and one that just repackages Rust functions and exports them into a Lua module. That isolates the CLI runtime with the Lua VM in one crate and a loadable Lua module in another. If running the application's Lua code base as a library loaded into Lua the Lua module will be able to provide all the expected API surface area.

So now we get to the actual question!

This layout has me creating two functions with two function signatures and hand converting the types for each side. For example in the POC of worked up, in the main crate's library we might have a function that returns a string:

pub fn demo() -> Result<String> {
    Ok("Hello from crate 1".to_string())
}

Simple enough. But we also need a similar function for the Lua module in the other crate:

use crate1;

fn demo(lua: &Lua, (): ()) -> LuaResult<LuaString> {
    let res = crate1::demo().unwrap();
    lua.create_string(res)
}

In this particular case it isn't too complex with no input argument types to convert and only a Result<String> to turn into a LuaResult<LuaString>, but I can see this getting more complex (or at least verbose) in a hurry. As soon as each function has a few different input args and possible multiple return arguments this is going to be some verbose casting boilerplate code.

Then we need to actually stuff that in a Lua module, so each function will also have to have an extra line in something like this:

#[mlua::lua_module]
fn mymodule(lua: &Lua) -> LuaResult<LuaTable> {
    let exports = lua.create_table().unwrap();
    let demo: LuaFunction = lua.create_function(demo).unwrap();
    exports.set("demo", demo)?;
    Ok(exports)
}

Is there something more ergonomic that I'm missing in any of this layout to perhaps do this in one crate, to somehow derive the right types for re-exporting to the Lua side, or automatically collate them in the module?

[khvzak]

Regarding the first point, functions like pub fn demo() -> Result<String> can be directly exported to Lua using Function::wrap(demo) OR Function::wrap_raw(demo) without writing much boilerplate since this commit.

[alerque]

Cool! One more reason to look forward to v0.10!

[alerque]

I just gave v0.10.0-beta.2 a spin, but it didn't work like you suggested. It did cleanup one line of code:

-let demo: LuaFunction = lua.create_function(demo).unwrap();
+let demo = Function::wrap(demo);

But what demo is here didn't get any less verbose. I tried passing it the fn demo() -> Result<String>, but it complained "expected function that takes 2 arguments". What it seems to want (and successfully builds with is still the messy fn demo(lua: &Lua, (): ()) -> LuaResult<LuaString> definition that gets the result from the actual Rust function, uwraps it, and converts it to a LuaString.

[khvzak]

The commit I referred to, was made after beta.2 version. You could try master or wait for rc.1 which will be approx on next week

[alerque]

Oh my bad, somehow I assumed it was older and just didn't even think to look at the beta tag point! I'm working on an experimental branch mostly so I might as well give master a shake.

[alerque]

I gave it a shake with Git HEAD (main branch), and got a little further. However I struggled a but to figure out how it was going to handle errors and the result does not seem satisfactory:

// Can NOT be wrapped (what I actually want)
type Result<T> = anyhow::Result<T>;
fn demo() -> Result<String>

// Can NOT be wrapped (what I tried next when anyhow failed)
fn demo() -> Result<String, std::error:Error>

// Can NOT be wrapped
fn demo() -> Result<String>

// Can be wrapped (but not useful as a Rust function outside of the Lua module wrapper)
fn demo() -> Result<String, LuaError>

Am I missing something obvious here?

[khvzak]

Can you try Function::wrap_raw ? The difference is - output type is not required to be result. if it's result, it will be converted to 2-values return (ok, err) rather than throwing an exception

[alerque]

That works to wrap fn demo() -> String, but that is not particularly useful since I need the Rust side of things to do proper error handling, not just unwrap everything. I had no luck using Function::wrap_wrap with any of the other forms that above that might be workable.

Would it be helpful to work up a MWE for this?

[khvzak]

Can you try the latest main branch with a new anyhow feature flag enabled. It enable IntoLua implementation for anyhow::Error. Then Function::wrap_raw should work.

[Calandiel]

Why are modules and vendored exclusive features anyway? Exposing an API for Lua scripts via modules is arguably the standard approach to bindings (see for example the sol library or OpenMW).

As it is, it seems like if you use the vendored feature you're encouraged to expose the API through global table assignment (this is also what all examples in the readme do), which is less than ideal as most existing tooling for lua expects things to be exposed with modules that can be required instead.

[khvzak]

@Calandiel I'm not sure I understand your question.

vendored feature flag is used to build Lua library from sources and statically link with the program. Lua modules should not do this and instead resolve symbols from the app that loads the module.

[alerque]

I'm not sure I understand @Calandiel's comment either, but there does seem to be a little bit of a catch-22. For example I have a project that both builds an app and a module (possibly for use together, but also possibly separate). Of course having the module linking from the host apps Lua VM makes sense (not static), but my host app does get built with vendored. That makes for a tricky build system where the app needs to get build and gets to use it's own vendored Lua sources, but building the module for use by the said-same app gets built separately against system supplied Lua headers. In my case I can make those match so the module works with the app too, but not being able to use the same vendored sources for both builds is kind of strange.

This whole topic should probably be a separate issue though, this is starting to wander from the initial topic which has been largely addressed already with v0.10.

[Calandiel]

@Calandiel I'm not sure I understand your question.

vendored feature flag is used to build Lua library from sources and statically link with the program. Lua modules should not do this and instead resolve symbols from the app that loads the module.

Could you specify which bit is problematic? It's difficult to clarify a message as a whole without knowing what's not clear ^^

I see Lua bindings through the lense of game development, maybe that explains the miscommunication.

In such projects you want to expose an API for modders to write scripts with, ideally without polluting the global namespace. I listed some existing projects using such architecture.

In essence, you want a script you can require by, say, local bindings = require 'bindings', where the bindings are a module defined by the host application. As I understand it, that isn't possible with vendored builds. Am I mistaken?

Let me know if that helps, but @alerque , I think, has a similar problem to mine :slightly_smiling_face:

[alerque]

@Calandiel Please open this as a new issue, you can use the top right menu on your first comment to "reference in new issue" and we can continue there.

[Calandiel]

@Calandiel Please open this as a new issue, you can use the top right menu on your first comment to "reference in new issue" and we can continue there.

Doesn't seem to me like it's necessarily a separate issue - to quote your original post:

One complicating factor is that this needs to compile in several modes: with or without being built against system Lua sources (so the vendored feature), and also dynamic or static linking, and with the static option also a variant that has all the Lua resources (script and C modules) embedded in the binary. All that works already, but we have two more layers before things start getting weird.

And the final note in it:

Is there something more ergonomic that I'm missing in any of this layout to perhaps do this in one crate, to somehow derive the right types for re-exporting to the Lua side, or automatically collate them in the module?

These are more or less my questions too :slightly_smiling_face:

[khvzak]

Just to clarify:

• mlua modules does not have more restrictions than Lua C modules. Both of them must not be statically linked with Lua library (what vendored feature does)
• You don't need to use globals to define module exports. Similar to Lua C modules, you can return any Lua value during module initialization ("require" stage)

In essence, you want a script you can require by, say, local bindings = require 'bindings', where the bindings are a module defined by the host application. As I understand it, that isn't possible with vendored builds. Am I mistaken?

Apps build in vendored mode can load any modules (C/Rust/Zig/etc). Modules cannot be build in vendored mode by Lua design and in fact there is not sense to do it (in any language).

[alerque]

• Both of them must not be statically linked with Lua library (what vendored feature does)

It seems like the vendored feature has been overloaded with two distinct modes rolled into one that should actually be separate. Static vs. dynamic linking for generated modules should not be the same thing as using system provided header files or vendored one during compilation.

[khvzak]

vendored builds are always static builds. Dynamic builds require having a loadable .so/.dylib file and outside of mlua scope to provide them. mlua does it's best to support any dynamic linking:

• automatically via pkg-config
• manually via LUA_LIB/LUA_LIB_NAME/LUA_LINK environment variables

Modules are not linked either statically or dynamically with Lua library. They use dynamic lookup for undefined symbols (see -undefined=dynamic_lookup option) at runtime ("dlopen" stage).

The fact that modules are not linked with Lua library allow any option for host apps: static builds and dynamic builds. Host apps can be dynamically linked with Lua library with any name, like lua.so, lua5.1.so, lua-5.1.so and so on.

PS Windows is a bit different and does not support dynamic lookup (I don't have much experience with windows and this area for me is a gray zone).