Byte code compilation

[splitice]

Looks good, I'd love our package management processes to get more mature. Many moons ago we developed a basic packaging system for use internally. Nothing at even this level, just something basic which we could use with ansible to version and deploy our code.

One thing we did is integrate bytecode compilation into our process. Effectively the way we did it was with a package step, that similarly to this produces compressed tar balls. Then on the relevant server an installation utility which extracts the code into a directory, then compiles each file, putting the output into a merged directory.

Any thoughts on this?

[agentzh]

@splitice Bytecode is not very portable (for example, LuaJIT 2.0, 2.1 and standard Lua 5.1 all use incompatible bytecode formats). So it does not make sense to do bytecode compilation in the packaging phase for opm (well, it may make sense to your own packaging toolchain since you usually just stick with a particular Lua/LuaJIT version in your own environment).

For opm, I think it only makes sense to do such bytecode compilation in the installation phase on the user side. But we still need to solve the following problems:

1. we need a separate tool or command to re-compile all the Lua source into a new bytecode format when the user upgrades the LuaJIT version (big version change, like from 2.1 to 2.2) in her OpenResty.
2. we need to add paths of *.luac to the standard Lua module search paths (package.path) unless we want to overwrite *.lua files with their bytecode (we do not want to lose the original source, since the user may need to re-compile it to the a bytecode format (see 1.) or just edit the Lua code to make her own changes either temporarily or permanently.

In general, bytecode compilation adds significant complexity and potential problems to the mix. That's why I won't add it any time soon. But it's good that we start talking about any viable solutions to it right now :)

[splitice]

I 100% agree, its an installation step for us. I probably didn't make it very clear.

The main reasoning behind bytecode compilation for us is to accelerate webserver startup & reload.

For us, its just an ansible script with options (lua_deploy_bytecode: True). But I imagine perhaps something like:

opm get --bytecode splitice/resty-awesomeness

The only thing then required is one final step to bytecode compile.

If its an optional flag, perhaps it becomes the users responsibility to worry about platform upgrades?

[agentzh]

@splitice Generate the byte code files as *.lua or *.luac? You didn't address the 2 problems stated in my previous comment, BTW.

[splitice]

More information.

1. If its an optional flag, perhaps it becomes the users responsibility to worry about platform upgrades? Thats my view, if your mature enough to need / want to use bytecode, your mature enough to know the effects and limitations.
2. We are using *.luac, the *.lua's of compiled files are kept on the server (currently) however are not at all in the search path.

Our structure is something like this:

packages/source/blah/lib/namespace/file.lua -> lib/namespace/file.luac

lib/ has all packages munged into one structure.

At one point lib was *.lua's maintained by symlinks. Now with bytecode in use, no symlinks.

[agentzh]

@splitice Seems like it is not worth adding into opm. It could be a separate tool that traverses the directory tree and compiles all the *.lua to *.luac, including Lua libraries already installed (like those shipped with OpenResty by default).

[agentzh]

@splitice Or maybe we could add a bytecode option to OpenResty's build system and OpenResty installs all its standard Lua modules as *.luac by default. The opm tool can honor that setting and always installs *.luac automatically. And OpenResty also always uses *.luac in its default package search paths.

[agentzh]

@splitice The LuaJIT bytecode mode can be an attribute of the user's OpenResty installation, which looks like the cleanest way.

[splitice]

I agree with the last two.

FYI its probably only a small addition either way, just seems like something useful.

[agentzh]

@splitice Yes, the implementation won't be difficult at all. Just a design issue. Pull requests welcome :)

[splitice]

Hehe, I think I've heard that before.