Benchmarked with lua-resty-json?

[Anorov]

Has this been benchmarked to compare its performance with lua-resty-json?

[bungle]

I just tried some benchmarks. All the tests were run with resty (nginx version: openresty/1.7.7.1rc2) (luajit enabled runtime).

Code is something like this (note: if you use this code, please split each test to different file and run them separately):

local open, clock = io.open, os.clock

local d1 = require "cjson".decode
local d2 = require "resty.libcjson".decode
local ins = require "json_decoder".new()
local d3 = ins.decode

local file = open("citylots.json", "rb")
local content = file:read "*a"
file:close()

local iterations, x, z = 1, 0, 0

x = clock()
for i = 1, iterations do
    d1(content)
end
z = clock() - x

print(string.format("D1 Decoding Time: %.6f", z))

x = clock()
for i = 1, iterations do
    d2(content)
end
z = clock() - x

print(string.format("D2 Decoding Time: %.6f", z))

x = clock()
for i = 1, iterations do
    d3(ins, content)
end
z = clock() - x

print(string.format("D3 Decoding Time: %.6f", z))

The citylots.lua was downloaded from here: https://raw.githubusercontent.com/zemirco/sf-city-lots-json/master/citylots.json

Here are the results:

D1 Decoding Time: 6.125711
D2 Decoding Time: 8.424276
nginx: lua atpanic: Lua VM crashed, reason: not enough memory

Another try:

D1 Decoding Time: 6.148450
D2 Decoding Time: 8.406922
D3 Decoding Time: 7.988033

I also tried smaller json-file (the web-app example from here: http://json.org/example) with 1000 iterations, and the results are here:

D1 Decoding Time: 0.028434
D2 Decoding Time: 0.086863
D3 Decoding Time: 0.026841

So, if we could have some conclusions here is that from a perfomance point of view:

1. lua-resty-json is the fastest, but can crash LuaJIT VM.
2. bundled lua cjson lib is almost as fast, but more reliable on larger files
3. lua-resty-libcjson is the slowest, but still reliable on larger files but takes forever to encode large files

Other differences:

1. lua-resty-json implements decoder only (others have decoder and encoder as well)
2. lua-resty-libcjson retains information about arrays and objects (not sure about others)

I also did small encoding BM between the bundled cjson lib and lua-resty-libcjson:

cjson results (using that citylots.json sample):

D1 Decoding Time: 6.206812
D1 Encoding Time: 4.894658

lua-resty-libcjson results:

D2 Decoding Time: 8.445740
D2 Encoding Time: (didn't wait for it to finnish)

The general advice:

1. Use the bundled cjson library (that is Lua CJSON: http://www.kyne.com.au/~mark/software/lua-cjson.php)
2. Use lua-resty-json if you need the last bits of performance (on their own page they say that it is roughly 30% faster)

Regards Aapo

[bungle]

I'm currently writing a new parser only lib with jsmn (http://zserge.com/jsmn.html). Let's see how it works out compared to others. It should be a really fast one, but there is always possibility to screw the performance on LuaJIT side.

[bungle]

I already did jsmn bindings (not yet published), but the best I got from it was (with that citylots.json):

Decoding Time: 7.467336

Actually, it seems it is pretty fast, but only when stream processing. Parse without decoding stuff to Lua table structures took just:

Decoding Time: 3.069683

Twice as fast as others here, but not so useful on Lua side, albeit a great option for stream processing JSON. I will look if you can do parsing in a single pass to get the full benefits of jsmn.

[bungle]

I also did some adjustments to lua-resty-libcjson code, and now it seems like the decoding performance is closer to bundled cjson:

lua-resty-cjson citylots decoding time: 6.536133

cjson citylots decoding time: 5.990543

[bungle]

I did rerun citylots test with each decoder separately:

1. cjson decoding time:              5.815602
2. lua-resty-libcjson decoding time: 6.349700
3. lua-resty-json decoding time:     7.014575
4. lua-resty-jsmn decoding time:     8.208794

Right now it seems that in this test the bundled Lua cjson library is the fastest, and the lua-resty-json is the slowest (lua-resty-jsmn decoding is not really optimized right now - its single pass parsing is the fastest).

[1] https://github.com/mpx/lua-cjson/ [2] https://github.com/bungle/lua-resty-libcjson [3] https://github.com/cloudflare/lua-resty-json [4] https://github.com/bungle/lua-resty-jsmn

[bungle]

I have now done some adjustments in lua-resty-jsmn[1] codebase, and this citylots test now gives this (using decode2 function, and jsmn lib compiled with -O3):

Decoding Time: 5.956955

It is now among the fastest.

[1] https://github.com/bungle/lua-resty-jsmn

[agentzh]

This is not what I'd expect because by swapping lua-cjson with lua-resty-json in CloudFlare's Lua CDN, the JSON decoding part almost disappears from our online (Lua-land) flame graphs and the reduction of the JSON decoding time can be as big as 80%. @yangshuxin will you have a look at this benchmark?

[bungle]

Let me add lua-resty-opjson (https://github.com/bungle/lua-resty-opjson) to this list as well (citylots):

Decoding Time: 5.820199

[bjoe2k4]

One should also consider encoding time. For me, lua-resty-libcjson performs really bad for encoding, which has probably something to do with the huge memory consumption while encoding (for files > 100KB i always get segfaults/coredumps). Since lua-resty-json has no encoder jet, cjson remains the only sane choice.

local open, clock = io.open, os.clock

local cjson = require "cjson"
local libcjson = require "resty.libcjson"
local d1 = cjson.decode
local e1 = cjson.encode
local d2 = libcjson.decode
local e2 = libcjson.encode
local ins = require "json_decoder".new()
local d3 = ins.decode

--http://mtgjson.com/json/pCEL.json
local file = open("/etc/nginx/pCEL.json", "rb")
local content = file:read "*a"
file:close()
local contentdata = d2(content)

local iterations, x, z = 100000, 0, 0

x = clock()
for i = 1, iterations do
    d1(content)
end
z = clock() - x

print(string.format("D1 Decoding Time: %.6f", z))

x = clock()
for i = 1, iterations do
    d2(content)
end
z = clock() - x

print(string.format("D2 Decoding Time: %.6f", z))

x = clock()
for i = 1, iterations do
    d3(ins, content)
end
z = clock() - x

print(string.format("D3 Decoding Time: %.6f", z))

x = clock()
for i = 1, iterations do
    e1(contentdata)
end
z = clock() - x

print(string.format("E1 Encoding Time: %.6f", z))


x = clock()
for i = 1, iterations do
    e2(contentdata)
end
z = clock() - x

print(string.format("E2 Encoding Time: %.6f", z))

D1 Decoding Time: 4.019130
D2 Decoding Time: 7.168249
D3 Decoding Time: 3.209868
E1 Encoding Time: 2.337952
E2 Encoding Time: 19.406382

[bungle]

@bjoe2k4 I agree with your conclusion. To make the built-in cjson even better one should consider writing it in pure LuaJIT, but that is enourmous amount of work. lua-resty-libcjson is more or less just a small wrapper to a libcjson C-library (it seems Lua cjson is somewhat a fork from that lib).