jsonlite
jsonlite copied to clipboard
[OPTIMIZATION] Replace base readLines with Kmisc::readlines
Have you considered replacing the read portion of your package with more optimal versions of the same functions?
I don't know the package well enough, but will make (breaking) modifications to stream_in in order to take advantage of the Rcpp implementation.
> microbenchmark::microbenchmark(base_readLines = base::readLines(file("/tmp/test.json")), Kmisc_readlines = Kmisc::readlines("/tmp/test.json"))
Unit: milliseconds
expr min lq mean median uq max neval cld
base_readLines 93.132934 96.942261 112.959125 102.232657 121.459780 216.66778 100 b
Kmisc_readlines 5.503058 6.085259 8.100304 6.460504 8.263181 29.45322 100 a
> stopifnot( identical(base::readLines(file("/tmp/test.json")), Kmisc::readlines("/tmp/test.json")))
>
Well it looks like most of the overhead is actually in post_process.
Have you looked at ways of speeding up this portion of the call?