Add option to limit the memory usage of Lua code

[Le0Developer]

Closes #211.

[Le0Developer]

It seems like LuaJIT does not support a custom allocator in lua_newstate on 64bit.

See the following and failing tests:

Make sure you use luaL_newstate. Avoid using lua_newstate, since this uses the (slower) default memory allocator from your system (no support for this on 64 bit architectures).

• https://luajit.org/install.html

[scoder]

It seems like LuaJIT does not support a custom allocator in lua_newstate on 64bit.

Hmm. That is annoying. Then we should raise an exception when we detect that case and users request a custom allocator.

[Le0Developer]

Setting max_memory to None will now use the default lua allocator. Also added documentation for set_max_memory() and max_memory.

set_max_memory() also a strict option now, which will raise an exception if you try to set max_memory to a value lower than currently used.

[Le0Developer]

Just tested running LuaJIT 2.1.0-beta3 and it works on 64bit. 🤷

Following is a patch that disallows LuaJIT on 64bit, if needed:

diff --git a/lupa/lua.pxd b/lupa/lua.pxd
index 35d4b6e15a4fc18e3deb20353be72cae4ebaee86..72cdce030bfaf13db38af6d97aed66c85f55eacf 100644
--- a/lupa/lua.pxd
+++ b/lupa/lua.pxd
@@ -266,6 +266,13 @@ cdef extern from "lua.h" nogil:
         int i_ci               #           active function */
 
 
+################################################################################
+# luajit.h
+################################################################################
+
+cdef extern from "luajit.h" nogil:
+    int LUAJIT_VERSION_NUM
+
 ################################################################################
 # lauxlib.h
 ################################################################################
@@ -465,11 +472,18 @@ cdef extern from * nogil:
     #if LUA_VERSION_NUM < 502
     #define lua_pushglobaltable(L)  lua_pushvalue(L, LUA_GLOBALSINDEX)
     #endif
+
+    #if defined LUAJIT_VERSION_NUM && UINTPTR_MAX == 0xffffffffffffffff
+    #define IS_LUAJIT64 1
+    #else
+    #define IS_LUAJIT64 0
+    #endif
     """
     int read_lua_version(lua_State *L)
     int lua_isinteger(lua_State *L, int idx)
     lua_Integer lua_tointegerx (lua_State *L, int idx, int *isnum)
     void lua_pushglobaltable (lua_State *L)
+    char IS_LUAJIT64
 
 cdef extern from *:
     # Limits for Lua integers (in Lua<5.3: PTRDIFF_MIN, PTRDIFF_MAX)
diff --git a/lupa/_lupa.pyx b/lupa/_lupa.pyx
index 22656680343b1c6e9f97ef8adf972df9958e614a..c173d9a9376339e2dfec982e04ca5ca9b7a8b6d2 100644
--- a/lupa/_lupa.pyx
+++ b/lupa/_lupa.pyx
@@ -263,6 +263,9 @@ cdef class LuaRuntime:
                   bint register_eval=True, bint unpack_returned_tuples=False,
                   bint register_builtins=True, overflow_handler=None,
                   max_memory=None):
+        if lua.IS_LUAJIT64 and max_memory is not None:
+            raise RuntimeError("max_memory is not supported on 64bit LuaJIT")
+        
         cdef lua_State* L
 
         self._memory_left = 0

[scoder]

set_max_memory() also a strict option now, which will raise an exception if you try to set max_memory to a value lower than currently used.

I'd rather expose the current memory usage as readonly attribute and let users do their own checks. That seems more generally useful.

[Le0Developer]

Added memory_used

[scoder]

There's still a test failure. Maybe you could look into that? I'd like to release Lupa 2.0 soon and this would be a nice addition.

[Le0Developer]

Unable to reproduce the test failure locally but I think that should've fixed it.

[Le0Developer]

I am getting no more test failures locally (running with 53 and 54)

[scoder]

How long will lupa still support py2? It's been EOL for over 1.5 years.

Which doesn't mean that people aren't using it any more. I'll support Py2 as long as it's easy to do, which it still is.

[Le0Developer]

Test failure seems unrelated and can't reproduce

[Le0Developer]

I'm a Cython noob, so no idea how to fix this 🤷

Error compiling Cython file:
------------------------------------------------------------
...
    cdef bytes _source_encoding
    cdef object _attribute_filter
    cdef object _attribute_getter
    cdef object _attribute_setter
    cdef bint _unpack_returned_tuples
    cdef MemoryStatus _memory_status
                     ^
------------------------------------------------------------

lupa/lua53.pyx:260:22: Variable type 'MemoryStatus' is incomplete

[Le0Developer]

All test succeeding locally now! 🎉

[scoder]

Thanks!