fromJSON is very slow in simplifyDataFrame function

[andrewkho]

Hi, I'm using a package which translates ElasticSearch queries (JSON format) to data.tables. The issue is that it's very slow, and majority of the time seems to spent in jsonlite's 'simplifyDataFrame' function.

If I call: jsonList <- jsonlite::fromJSON(m, flatten=T)

where m is a 17.9 Mb json string, it takes over 35 seconds, resulting in a 120.8 Mb object (jsonList). I ran it through Rprof and it looks like most of the bottleneck is in simplifyDataFrame. Here's the first few rows of the profiling summary, I'll also attach the raw profile data:

$by.total
                           total.time total.pct self.time self.pct
"chomp_aggs"                     44.72     99.91      0.00     0.00
"lapply"                         42.98     96.02      2.68     5.99
"FUN"                            42.92     95.89      4.76    10.63
"fromJSON_string"                35.86     80.12      0.00     0.00
"jsonlite::fromJSON"             35.86     80.12      0.00     0.00
"simplifyDataFrame"              35.02     78.24      2.00     4.47
"simplify"                       35.02     78.24      0.00     0.00
"vapply"                          9.56     21.36      3.72     8.31
"unpack_nested_data"              8.52     19.03      0.00     0.00
"unlist"                          7.44     16.62      1.34     2.99
"data.frame"                      4.94     11.04      0.84     1.88
"unique"                          4.58     10.23      1.78     3.98
"as.data.table.data.frame"        4.00      8.94      0.12     0.27

Is there anyway this method could be sped up?

[exaexa]

+1, same issue (triggered by pushing large amounts of data to browsers from Shiny)

Is there any update since the bug was opened?