High memory usage using 'push_variant' to send an array to lua

[kicknbrit]

So after some investigation, I found pushing a 7MB PackedByteArray to lua takes up about 8 GB of memory on my machine.

I allocated the same size array in lua (zerobranestudio) and it took only around 141 MB.

Here is the code I am using:

func _ready():
    
    lua = Lua.new()
    
    var buffer = PackedByteArray()
    buffer.resize(7000000)
    buffer.fill(0)
    buffer[77] = 55
    
    lua.push_variant(buffer, "buffer")
    lua.do_string("print(buffer[78])")
    
    
    
    pass

[Trey2k]

Seems like some sort of memory leak from the description. I will be looking into this over the weekend. I will keep you updated.

[kicknbrit]

Just out of curiosity, does push_variant create new lua tables and copy the data from the PackedByteArray to the new tables?

[Trey2k]

Just out of curiosity, does push_variant create new lua tables and copy the data from the PackedByteArray to the new tables?

It's situational. But typically the answer is yes. Especially for primitive types. Godot did not like to shsre its memory. There probably needs to be some major overhaul to the way memory is handled currently. But yes, currently there is extensive use of memcpy.

[kicknbrit]

I read somewhere recently about godot's memory being fickle. It's unfortunate. However, I think your current implementation is fine.

[Trey2k]

Update - I am still searching for the cause. I wrote a small test program. I can confirm this code only takes up 193MB of ram.

#include <lua.h>
#include <lualib.h>
#include <lauxlib.h>

int main() {
    lua_State *L;
    L = luaL_newstate();

    lua_createtable(L, 0, 7000000);

    for (int i = 0; i < 7000000; i++) {
        long long key = i+1;
        lua_pushinteger(L, key);

        if (i==77) {
            lua_pushinteger(L, 55);
        } else {
            lua_pushinteger(L, 0);
        }
        
        lua_settable(L, -3);
    }

    lua_setglobal(L, "buffer");

    luaL_dostring(L, "print(buffer[78])");

    lua_close(L); 

    return 0;
}

That code is pretty much doing the exact thing this module does. This is where we handle arrays being pushed to lua: https://github.com/WeaselGames/lua/blob/ce62c1182c224bdf29ea0fcfdd42403090280c46/lua.cpp#L184-L205 Which calls push_variant again for the key and value. Which in your case above would both be the type INT. This is where INT is handled: https://github.com/WeaselGames/lua/blob/ce62c1182c224bdf29ea0fcfdd42403090280c46/lua.cpp#L175-L177

My only thought is lua is not causing the memory leak but some other interaction with this module and godot is. Will need to look further.

[Trey2k]

Small note its not a solution but if we replace pushVariant on 200 and 201 with lua_pushinteger. Memory usage plumets. So it looks like some aspect of the recursive call is causing the leak. I am still at a loss though.

[Trey2k]

I believe the issue is the fact that pushVariant returns a LuaError::errNone() which is not a null pointer and does use a memnew call. For the nested recursive calls of pushVariant we currently do not capture and free the return value. And I am sure there is other places it has not been handled as well. This is the cause of the leak.

[Trey2k]

Fixed in 65d5799178b1a6d973466650ed827ea4b0c06b0a