JSON.jl

JSON.jl copied to clipboard

@samuelcolvin mentioned that he thought Julia's JSON code is slow right now. I wasn't aware of any egregious performance problems, but it's true that it probably hasn't been much optimized.

It would be nice to do some performance benchmarks against C and C++ JSON implementations, e.g UltraJSON, JsonCpp, Jansson, YAJL, and JSON Spirit.

Nothing like being put on the spot for a side note in completely different discussion :-).

As it happens I have some results from a similar comparison. It's between Python's JSON parsing/generating and Julia, not C or C++.

Still I think the results are interesting:

Python results:

generating data...
performing tests...
python json dump time:    4.063s, len = 48.11m
python ujson dump time:   1.465s, len = 42.74m
python json load time:    3.004s
python ujson load time:   1.012s

This generates a json file 40-50mb in size.

Julia results:

file loaded and read:            elapsed time: 1.177662644 seconds
Parsing JSON from string:        elapsed time: 25.780348228 seconds (831651540 bytes allocated)
Parsing JSON directly from file: elapsed time: 61.440253927 seconds (1012071980 bytes allocated)
Generating JSON:                 elapsed time: 7.397358822 seconds (211069604 bytes allocated)

Not impressive performance from Julia I'm afraid - tell me I'm doing something very dump wrong? parsing an IO object in particular should should be easy to improve the performance of.

Code is in a gist here. I'm aware that it seem odd to use Python to generate the test dataset when we're interested in Julia, but that's what I had and we will probably need a better dataset anyway - see below.

Disclamer: I'm aware that a bunch of things could have a significant affect on performance (and relative performance) notably:

1. Data type.
2. Encoding.
3. Data shape.
4. Strange escape sequences???
5. Reading direct form disc vs. from memory.

This is almost certainly not the best data set to benchmark or compare libraries with. However I don't know of a better one, and it shows similar results to those I've found when using Python and Julia's JSON libraries elsewhere.

This is not too surprising. The JSON package has not been optimized much. We should make heavily optimizing it a priority and a test case – we need to make some changes to our I/O subsystems to allow better performance in general and then make use of them in JSON.jl.

Well, if its any consolation (probably not!), JSON parsing was even slower in its original incarnation. Dirk rewrote the parser to be much faster than it used to be.

It looks like the parsing makes very heavy use of memory allocation, with lots of temporary objects. This is currently slow in Julia, although there are pending improvements to the Julia allocator/gc that should help. But it looks fairly straightforward to re-write the routines to re-use buffers, and I see lots of other low-hanging optimization opportunities in a quick glance over the code.

From a quick read, it looks to me like the JSON parser shares a lot of performance problems with readtable in DataFrames.jl. Coming up with better tools in Julia for doing things like parsing numbers without allocating temporary strings would make the DataFrames code easier to maintain in addition to improving JSON parsing.

I've added some results for a JSON file with no strings. Sure enough there's significant speed up and relative speed up:

Parsing:
    Julia: 3.8 x faster
    Python: 1.9 x faster
Generating:
    Julia: 2.46 x faster
    Python: 1.2 x faster

(Part of the speed up is because the JSON file is slightly smaller than before I guess.)

This is consistent with serialize and deserialize where stings make a massive different to performance.

As a slightly different approach I started an alterative JSON library based on ultra JSON, it's at uJSON.jl.

Currently it's just parsing not printing but it's significantly (6x) faster with arrays and 1.5x faster with a mixture of arrays and Dicts than the standard JSON library. It's also much faster at parsing JSON with lots of levels. Obviously the relative performance varies but it seems generally quicker even without much optimization.

With dicts setindex! seems to take up a big chunk of the time and since both libraries use it both their performances are much worse.

Isn't this a the kind of application where we should try to benefit from someone else's hard work?

@samuelcolvin, will you be submitting uJSON.jl to METADATA?

For work, I would love to use uJSON.jl. At the same time, I would love to see JSON.jl improved as much as possible. It's easier to install (no binary dependency), and provides a good test where Julia itself could use some improvements.

I can do, I ended up unsure that the performance improvements were significant enough to make it worthwhile.

I was also intending to add print functionality as well as parse but haven't got around to it. Maybe that's not really worth it, most of the "slow" stuff in print (eg. interrogating variables) will have to stay in Julia so a c implementation wouldn't result in noticable performance improvements.

I'll try and look at it this week.

Just came back to this and I'm please to see how much JSON.jl's performance has improved, however it's still slower than ujson. What would be the best approach to equalling it's performance?

samuel:11101522 || python json_performance.py 
2.7.8 |Anaconda 2.0.1 (64-bit)| (default, Aug 21 2014, 18:22:21) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
generating data...
performing tests...
python json dumps time:   1.193s, len = 47.66m
python ujson dumps time:  0.218s, len = 42.36m
python json loads time:   0.914s
python ujson loads time:  0.305s
samuel:11101522 || ls -lh
total 46M
-rw-rw-r-- 1 samuel samuel  418 Oct  4 14:19 json_performance.jl
-rw-rw-r-- 1 samuel samuel 1.1K Oct  4 14:21 json_performance.py
-rw-rw-r-- 1 samuel samuel 1.9K Oct  4 14:19 results.md
-rw-rw-r-- 1 samuel samuel  46M Oct  4 14:22 test_data.json

with Julia 0.3:

samuel:11101522 || julia json_performance.jl 
Julia Version 0.3.2-pre+41
Commit 895639a* (2014-10-03 18:51 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
file loaded and read:            elapsed time: 0.09773618 seconds
Parsing JSON from string:        elapsed time: 2.761333869 seconds (686891224 bytes allocated, 33.73% gc time)
Parsing JSON directly from file: elapsed time: 4.651523492 seconds (856648424 bytes allocated, 10.36% gc time)
Generating JSON:                 elapsed time: 1.284359838 seconds (277870740 bytes allocated)

with Julia 0.4:

samuel:11101522 || julia json_performance.jl 
Julia Version 0.4.0-dev+940
Commit 265f9b8 (2014-10-04 10:37 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
file loaded and read:            elapsed time: 0.081438187 seconds
Parsing JSON from string:        elapsed time: 1.960918887 seconds (681979904 bytes allocated, 1.27% gc time)
Parsing JSON directly from file: elapsed time: 4.461541554 seconds (856509972 bytes allocated, 9.85% gc time)
Generating JSON:                 elapsed time: 2.354674854 seconds (526854656 bytes allocated)

Can we update this in light of #140 by @TotalVerb?

With the current JSON.jl and Julia 0.4, I get:

$ python json_performance.py 
2.7.10 |Anaconda 2.3.0 (x86_64)| (default, Oct 19 2015, 18:31:17) 
[GCC 4.2.1 (Apple Inc. build 5577)]
generating data...
performing tests...
python json dumps time:   1.429s, len = 27.04m
python ujson dumps time:  0.189s, len = 21.69m
python json loads time:   0.652s
python ujson loads time:  0.193s

$ julia-0.4 json_performance.jl 
Julia Version 0.4.3
Commit a2f713d (2016-01-12 21:37 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
file loaded and read:            elapsed time: 0.078572452 seconds
Parsing JSON from string:          0.920853 seconds (1.32 M allocations: 73.046 MB, 6.56% gc time)
Parsing JSON directly from file:   1.358616 seconds (9.93 M allocations: 429.248 MB, 17.17% gc time)
Generating JSON:                   0.866724 seconds (6.26 M allocations: 238.517 MB, 6.53% gc time)

Julia master is pretty similar.

These particular benchmarks are really number-heavy, which kills JSON.jl's performance. Our parse_number is type-unstable, and type-instability seems to cost more in Julia than in Python. Also, our strtod (which is a ccall to a Base function) is a little slow for some reason. I couldn't figure out how to remove the excess allocation in this function. On other benchmarks, I think we perform better than Python's json but worse than ujson.

It would be interesting to try this again on 0.6 with the new faster String implementation. I was looking at @samuelcolvin's gists above but couldn't figure out where the test data was from.

Haven't looked at this for some time, but it appears the test data is generated by json_performance.py, you just need to run that before json_performance.jl

When I profiled these benchmarks a while ago, the majority of the time was spent on parsing numbers, which was due to a combination of the type instability of parse_number and the performance of strtod. Perhaps the improved Union type handling in 0.6 has improved matters.

Any update on the most recent benchmarks?

You can run the benchmarks yourself using the script in bench/. If I get the time I will see if we can hook up PkgBenchmark

However, as there seems to still be a misconception: the string parsing was fine before the 0.6 optimizations, so I don't anticipate improvement. Number parsing is still the bottleneck on Samuel's benchmarks

FWIW, unless you want to use the macro based API in PkgBenchmark, "hooking up PkgBenchmark" will actually just mean to define a benchmark suite from BenchmarkTools so there is no need to depend on PkgBenchmark for it.

Ah, excellent. We already have a benchmark tools suite. We can expand it gradually then and use PkgBenchmark to track progress.

Unfortunately, the JSON that I need to parse is rather large and is a combination of strings and numbers. It might be worth standing up my own benchmarks for my specific use case.

Sounds good. If you can contribute an example benchmark to this project, that would also help the future optimization of JSON.jl.

The data itself is proprietary, but I could probably mock something up.

One other question I had is whether there would be a way to get a speed boost if the JSON has a known schema ahead of time?

Unless the schema is quite simple, no. The vast majority of the overhead is in parsing numbers (which does not get easier by knowing that it is a number, since numbers are fairly easy to identify) or in constructing dictionaries.

FWIW, parsing large amounts of Twitter data like this and this is about 2x slower than stdlib json in CPython.

Here is an example. Fetch a JSON-formatted array of 100k Twitter user objects. (In this case, it's just my own twitter user object repeated 100k times.):

wget https://freeboson.org/100k_twitter_users.json.bz2 -O- | bzcat > 100k_twitter_users.json

Parsing with stdlib json in python3:

$ python
Python 3.6.2 (default, Jul 20 2017, 03:52:27) 
[GCC 7.1.1 20170630] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from timeit import default_timer as timer; import json
>>> i=timer(); x=json.load(open("100k_twitter_users.json")); f=timer(); print("  {:6f} seconds".format(f-i))
  4.308163 seconds

Parsing with Julia / JSON.jl:

$ julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0 (2017-06-19 13:05 UTC)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |  x86_64-pc-linux-gnu

julia> using JSON

julia> @time x=JSON.parsefile("100k_twitter_users.json");
  8.408370 seconds (28.95 M allocations: 2.603 GiB, 9.91% gc time)

I will also note that the resident set size for julia got quite a bit larger than python.

Thanks for this benchmark. How much of this is compilation time? We won't really be able to do much about that part.

On the other hand it looks like there's more improvement possible here.

@TotalVerb it's negligible. The compilation time is amortized and the Julia/Python ratio of ~2 is consistent with yet larger arrays.

Related: https://discourse.julialang.org/t/announce-a-different-way-to-read-json-data-lazyjson-jl/9046