ffi.typeof limit and shm.lua

[mestorsi]

Hello,

I tried to create and open objects with shm many times аnd found the problem (function map from shm.lua - return ffi.cast(ffi.typeof("$&", ctype), mem)) which I tried to reproduce in the following code:

module(..., package.seeall)
local ffi = require "ffi"
function run()
    for i = 1,10000 do
        print("i = " .. tostring(i))
        for n = 0,9999 do
            local type = "struct {uint8_t value[" .. tostring(i*10000 + n) .. "];}"
            local ctype = ffi.typeof(type)
        end
        ffi.C.sleep(1)
    end
end

Result:

./snabb testtypeof
i = 1
i = 2
i = 3
table overflow

Stack Traceback
===============
(1) Lua function 'handler' at file 'core/main.lua:168' (best guess)
        Local variables:
         reason = string: "table overflow"
         (*temporary) = C function: print
(2) field C function 'typeof'
(3) Lua field 'run' at file 'program/testtypeof/testtypeof.lua:10'
        Local variables:
         (for index) = number: 3
         (for limit) = number: 10000
         (for step) = number: 1
         i = number: 3
         (for index) = number: 1373
         (for limit) = number: 9999
         (for step) = number: 1
         n = number: 1373
         type = string: "struct {uint8_t value[31373];}"
(4) Lua function 'main' at file 'core/main.lua:67' (best guess)
        Local variables:
         program = string: "testtypeof"
         args = table: 0x41754ac0  {}
(5) global C function 'xpcall'
(6) main chunk of file 'core/main.lua' at line 238
(7)  C function 'require'
(8) global C function 'pcall'
(9) main chunk of file 'core/startup.lua' at line 3
(10) global C function 'require'
(11) main chunk of [string "require "core.startup""] at line 1
        nil

Of course, after create() and open() I run unmap(), unlink() for every object.

[lukego]

The FFI is limited to defining max 64K types. Each time you call ffi.typeof("struct ...") you are creating at least one new type (and possibly more for pointers and references.) So it makes sense that after some tens of thousands of iterations this loop will exhaust all of the available 16-bit "ctype-id" values and you will get an error when trying to define a new type.

You probably need to reformulate your program to define fewer types and reuse them. This means at least caching the results of your calls to ffi.typeof(...). You might also find that you only need one type e.g. that you can move the object size from the type into the value with struct { int size; uint8_t values[0]; }.

I don't think this issue touches on operating system shared memory issues -- it's the Lua runtime.

Does that help?

[mestorsi]

Thanks for the reply, I will try your recommendation. I think struct { int size; uint8_t values[0]; } should help.

[lukego]

Just remember that every time you call ffi.typeof("struct ...") you will define a new C type, even if the structure definition is the same. So if you only need one type then be careful to only call ffi.typeof() once and to reuse the type object that it returns.