skynet icon indicating copy to clipboard operation
skynet copied to clipboard

sharedata got stack overflow when 100w string key in use

Open spin6lock opened this issue 1 year ago • 2 comments

--examples/sharedata_size.lua
local skynet = require "skynet"
local sharedata = require "skynet.sharedata"

local big_table = {}
local size = 1000000
for i=1,size do
        big_table[tostring(i)] = i
end

skynet.start(function()
        skynet.error("new foobar")
        sharedata.new("foobar", big_table)
end)

跑这个样例的时候会报错StackOverflow

[:01000004] master listen socket 0.0.0.0:2013
[:01000005] LAUNCH snlua cslave
[:01000005] slave connect to master 127.0.0.1:2013
[:01000004] connect from 127.0.0.1:33644 4
[:01000006] LAUNCH harbor 1 16777221
[:01000004] Harbor 1 (fd=4) report 127.0.0.1:2526
[:01000005] Waiting for 0 harbors
[:01000005] Shakehand ready
[:01000007] LAUNCH snlua datacenterd
[:01000008] LAUNCH snlua service_mgr
[:01000009] LAUNCH snlua sharedata_size
[:01000009] Memory warning 32.00 M
[:01000009] Memory warning 64.00 M
[:0100000a] LAUNCH snlua sharedatad
[:01000009] new foobar
[:0100000a] Memory warning 32.00 M
[:0100000a] Memory warning 64.00 M
[:0100000a] lua call [1000009 to :100000a : 3 msgsz = 11757584] error : ./lualib/skynet.lua:988: ./lualib/skynet.lua:452: stack overflow
stack traceback:
        [C]: in function 'skynet.sharedata.core.new'
        ./service/sharedatad.lua:15: in upvalue 'newobj'
        ./service/sharedatad.lua:76: in local 'f'
        ./service/sharedatad.lua:162: in upvalue 'f'
        ./lualib/skynet.lua:402: in function <./lualib/skynet.lua:374>
stack traceback:
        [C]: in function 'assert'
        ./lualib/skynet.lua:988: in function 'skynet.dispatch_message'
[:01000009] init service failed: ./lualib/skynet.lua:720: call failed
stack traceback:
        [C]: in function 'error'
        ./lualib/skynet.lua:720: in upvalue 'yield_call'
        ./lualib/skynet.lua:737: in function 'skynet.call'
        ./lualib/skynet/sharedata.lua:45: in function 'skynet.sharedata.new'
        ./examples/sharedata_size.lua:12: in upvalue 'start'
        ./lualib/skynet.lua:1065: in function <./lualib/skynet.lua:1063>
        [C]: in function 'xpcall'
        ./lualib/skynet.lua:1067: in function 'skynet.init_service'
        ./lualib/skynet.lua:1080: in upvalue 'f'
        ./lualib/skynet.lua:375: in function <./lualib/skynet.lua:374>
[:01000009] KILL self
[:01000002] KILL self

注释掉 lualib-src/lua-sharedata.c:373能好,luaL_checkstack应该在使用lua栈之前用;但是有同学觉得lualib-src/lua-sharedata.c:383的lua_settop会导致栈顶设置异常,大家有什么建议呢?

spin6lock avatar Oct 08 '24 14:10 spin6lock

这里设计上是用 lua stack 当数组使用来保存一个 id 到 string 的映射表。并没有考虑超过 100 万的情况。 如果不重新实现,那么你需要修改 luaconf.h 里的 LUAI_MAXSTACK ,默认就是 100 万。

否者,你得重新实现这个映射表。比较接近的方案是使用 userdata 的 uservalue 来保存这张表,应该有更大的限制。 但 lua_getiuservalue 会略微慢于 lua_value

即 https://github.com/cloudwu/skynet/blob/master/lualib-src/lua-sharedata.c#L412 这里在 L 里面放一个足够大的 userdata 。然后用 lua_setiuservalue 把 string 放进去。

或者用一个 table 也可以,lua_rawgeti 会比 lua_getiuservalue 更慢一丁点。用 table 的好处是容量和 stack 一样是弹性的。userdata 的 uservalue 个数创建时就需要决定好。

cloudwu avatar Oct 09 '24 00:10 cloudwu

我们先分离下配置表,搞多几个入口

spin6lock avatar Oct 09 '24 03:10 spin6lock