Dynamically modifiable code

[Hawk777]

In some situations, it would be nice if we could execute code that was provided dynamically, at runtime, rather than only code statically burnt into the LuaController.

Two possible ways I can see doing this:

1. Expose an API function that allows writing to the code metadata entry at runtime.
2. Expose a wrapped version of loadstring that configures an appropriate environment and doesn’t permit loading bytecode. Of the two, I prefer option 1, as it means not having to spend time bringing in the code and loading it on every run.

This would be much like a real-life microcontroller having self-programming ability for its Flash memory, which most do these days. It could be very useful in a multi-processor system; all nodes could run identical code with some configuration data stored in mem to differentiate the nodes from each other. Firmware upgrades could then be loaded into one node and automatically distributed over the network to other nodes for installation, rather than having to manually visit each node and install the firmware by hand. For that matter, nodes could even run non-identical code, with a central “update server” which gets loaded with a package containing multiple firmware images which are then distributed to nodes as needed.

If option 1 happens, you probably want to both limit how large the code can be and also perhaps impose some sort of high cost for using this function to prevent people from abusing it as a second, larger form of mem. As an example, perhaps the LuaController stops working for one second after writing to code. This would be somewhat realistic, as real microcontrollers often take quite a while to write to Flash and sometimes stop executing instructions while doing so, depending on the specific model and organization into memory areas.

[Jeija]

https://github.com/minetest-mods/homedecor_modpack/pull/317#issuecomment-241247348 might also be relevant to this discussion. Having the sandbox abstracted away will make this easier. Also, the long term plan is to have a web-luacontroller, which should make this kind of feature a lot more interesting (see link above).

[Hawk777]

That’s an interesting proposal, but I think extracting the sandbox into a separate mod is pretty much orthogonal to this proposal. Yes, they both affect the sandbox, but moving it into a separate mod is pretty much a matter of cutting all the sandbox code out of luacontroller and pasting it into another file, while this proposal would just be about adding another function to the sandbox. In case 1 this would obviously have to be provided by the LuaController itself (and I think it’s pretty much guaranteed that you will always have some sandbox functions provided by the individual mod rather than the core sandbox—for example things that muck with Mesecons wires would be provided by the LuaController and won’t exist in the Home Decor computer), and then it would just be one more API that LuaController needs to register with sandbox once sandbox exists. In case 2, it would make sense for the sandbox core to expose loadstring, and it’s just one more function to cut and paste over into sandbox core.

As far as the Web-accessible LuaController is concerned, that would definitely be a helpful thing to have, and I’m pretty sure that, given a sane implementation, I would use it.

Is there a ticket open for the job of extracting the sandbox code? If not, should I open one? I think I could take a stab at implementing it if you like, but I’m not sure how you want to play things if (as I suspect) it would be a completely separate mod rather than part of Mesecons.

[Hawk777]

Did any progress ever happen on the idea of moving the sandboxing mechanism out of the Luacontroller layer? I’m interested in working on that if it’s still a project that you think is worth having. Would you like me to open a new issue where we can discuss plans? It seems that some duplication has shown up (the Lua-controlled tubes in Pipeworks have copied the sandboxing code), so IMO it would be worth doing the refactor. There are also some optimizations to the sandbox I want to propose, but I think it makes sense to do them after the refactor if we’re going to do it at all.

Thoughts?

[numberZero]

moving the sandboxing mechanism out of the Luacontroller

I think that does worth it.

Dynamically modifiable code

Controversial IMO.

[Hawk777]

Cool. Two questions then.

First, what’s the plan technically? Right now the same file handles the following jobs:

1. Defining a node.
2. Defining the formspec.
3. Migrating state (script source code and mem contents) into and out of the node.
4. Acting as a Mesecons effector and receptor.
5. Acting as a Digilines effector and receptor.
6. Providing the internal APIs that only apply within the scripting environment (heat, heat_max, mem, print, interrupt, string, math, table, and os).
7. Providing the external APIs that interact with the rest of the world (pin, port, digiline_send).
8. Providing a secure execution environment in which to parse and run the code (checking for binary input to loadstring, turning off JIT, setting up the timeout hook, building an appropriate environment).

I think it makes sense for each individual mod to do 1, 2, 3, 4, maybe 5, and most of 7. I think it makes sense for the sandbox to provide 6 and 8, and maybe some helpers for 3. Digilines I’m not sure, I could go either way: either it’s the individual mod’s responsibility, or perhaps we provide a mechanism by which a mod can ask to participate in Digilines (and provide a node position), and then the sandbox framework provides the necessary glue. Mesecons is more mod-specific (e.g. the Luacontroller has four ports named a, b, c, and d, while the Lua tube has six ports named for colours), so I don’t think the sandbox mod can usefully provide that.

The reason why I think 3 should be the individual mod’s responsibility (though the sandbox mod can maybe provide some helpers) is because the sandbox should be modular enough that someone could use it for e.g. a handheld computer, which doesn’t even have a node position at all and stores all its information in an item stack instead. So that would have different state storage requirements.

Thoughts on this part?

Second, what’s the plan for ownership? I’m not a member of the minetest-mods Github group, so I can’t create new repositories under that group. Should someone who is a member create an empty repo and then I submit a pull request to fill it? Or should I create the repo under my own name and then somehow transfer ownership? I don’t think I can transfer ownership without being a member of the group, but maybe there’s a way. Anyway I think if I do that it leaves behind a redirect, and would that then break things if I later wanted to fork the mod to submit more PRs (when my fork tried to overwrite the redirect)? I’m not sure how Github works in this regard.

[numberZero]

@Hawk777 https://github.com/numberZero/libluabox/invitations