jsonlite icon indicating copy to clipboard operation
jsonlite copied to clipboard

[OPTIMIZATION] Replace base readLines with Kmisc::readlines

Open ckatsulis opened this issue 7 years ago • 1 comments

Have you considered replacing the read portion of your package with more optimal versions of the same functions?

I don't know the package well enough, but will make (breaking) modifications to stream_in in order to take advantage of the Rcpp implementation.

> microbenchmark::microbenchmark(base_readLines = base::readLines(file("/tmp/test.json")), Kmisc_readlines = Kmisc::readlines("/tmp/test.json"))
Unit: milliseconds
            expr       min        lq       mean     median         uq       max neval cld
  base_readLines 93.132934 96.942261 112.959125 102.232657 121.459780 216.66778   100   b
 Kmisc_readlines  5.503058  6.085259   8.100304   6.460504   8.263181  29.45322   100  a 

>  stopifnot( identical(base::readLines(file("/tmp/test.json")), Kmisc::readlines("/tmp/test.json")))
>  

ckatsulis avatar Aug 16 '17 05:08 ckatsulis

Well it looks like most of the overhead is actually in post_process. image

Have you looked at ways of speeding up this portion of the call?

ckatsulis avatar Aug 16 '17 06:08 ckatsulis