Invalid or unaligned stack was encountered during an unwind operation

[fre-sch]

Using Visual Studio 2019, Windows 10, lua 5.3.5, sol2 (develop 2020-03-29)

In a project where I use sol2 in a DLL that's loaded rather unconventionally (MemoryModule https://github.com/fancycode/MemoryModule), I encountered:

Unhandled exception at 0x00007FFE5993BF98 (ntdll.dll) in test-exe-with-dll.exe: 0xC0000028: An invalid or unaligned stack was encountered during an unwind operation.

The code where this occurs:

    sol::state lua{};
    lua["echo"] = &echo;

    sol::environment env = sol::environment(lua, sol::create, lua.globals());
    lua.safe_script_file("scripts/script.lua", env);
    sol::protected_function hook = env["Hook"];
    hook();

I've created a VS2019 project on github to investigate this: https://github.com/fre-sch/test-exe-with-dll

Within this project, there's two DLLs generated, and both use the same lua as static library:

• foolib: using plain lua
• barlib: using sol2

Then within test-exe-with-dll I've prepared loading these DLLs:

• load_dofoo() loads foolib using LoadLibraryA and works
• load_dobar() loads barlib using LoadLibraryA and works.
• loadmm_dofoo() loads foolib using MemoryModule and works.
• loadmm_dobar() loads barlib using MemoryModule and produces the stack error.

Unfortunately I cannot use LoadLibraryA in the specific situation (DLL injection for game mods), and I'm not sure if there are alternatives to using MemoryModule, as that's outside of my control.

I've looked through the code how sol2 is using base lua in this situation, and that's how I pieced together the base lua version in my example. But I did not figure out why the base lua version is working where sol2 fails.

[fre-sch]

I've further investigated and updated my example:

• I made sure Lua is build as C++, as it really wasn't before. This changed the unaligned stack error to an exception within KernelBase.dll.
• I've defined SOL_PRINT_ERRORS=1
• Copied lines from safe_script_file, but used unsafe_function and unsafe_function_result instead of stack_aligned_protected_function and protected_function_result

Now sol prints an error: [sol3] An error occurred and panic has been invoked: scripts/script.lua:1: attempt to call a nil value (global 'echo'). Which looks like somehow the environment metatable __index isn't set to the globals table where I've added the echo function. When I do set the environment var echo to the function, no exception in KernelBase.dll is raised.

Now with env["echo"] set, if I use stack_aligned_protected_function as state_view::do_file does, the exception in KernelBase.dll is raised again.

I wouldn't mind copying what state_view::do_file does, to change from stack_aligned_protected_function to just protected_function but not being able to setup the environment to use the globals as metatable __index is problematic.

[ThePhD]

I'm taking a look at this now that I've got a bit of time.

[ThePhD]

Looks related to #965 w.r.t errors and functions on the stack and things push on them...