quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

use serde_json_borrow to parse JSON

Open PSeitz opened this issue 1 year ago • 1 comments

  • update tantivy
  • use serde_json_borrow instead of serde_json::Value
➜  quickwit-indices cat mezmo/mezmo-use-stage-2023-01-20-ndjson/lines.njson | quickwit tool local-ingest --index mezmo
serde_json_borrow + CompactDoc
 Num docs   46475 Parse errs     0 PublSplits   0 Input size    56MB Thrghput 28.42MB/s Time 00:00:02
 Num docs   94511 Parse errs     0 PublSplits   0 Input size   115MB Thrghput 38.54MB/s Time 00:00:03
 Num docs  138769 Parse errs     0 PublSplits   0 Input size   170MB Thrghput 42.67MB/s Time 00:00:04
 Num docs  184161 Parse errs     0 PublSplits   0 Input size   225MB Thrghput 45.15MB/s Time 00:00:05
 Num docs  227811 Parse errs     0 PublSplits   0 Input size   280MB Thrghput 56.02MB/s Time 00:00:06
 Num docs  273217 Parse errs     0 PublSplits   0 Input size   337MB Thrghput 55.50MB/s Time 00:00:07
 Num docs  320748 Parse errs     0 PublSplits   0 Input size   395MB Thrghput 56.17MB/s Time 00:00:08
 Num docs  366026 Parse errs     0 PublSplits   0 Input size   451MB Thrghput 56.45MB/s Time 00:00:09
 Num docs  410292 Parse errs     0 PublSplits   0 Input size   506MB Thrghput 56.45MB/s Time 00:00:10
 Num docs  456037 Parse errs     0 PublSplits   0 Input size   562MB Thrghput 56.26MB/s Time 00:00:11
 Num docs  501879 Parse errs     0 PublSplits   0 Input size   619MB Thrghput 56.09MB/s Time 00:00:12
 Num docs  549379 Parse errs     0 PublSplits   0 Input size   677MB Thrghput 56.47MB/s Time 00:00:13
 Num docs  594425 Parse errs     0 PublSplits   0 Input size   732MB Thrghput 56.45MB/s Time 00:00:14
 Num docs  638732 Parse errs     0 PublSplits   0 Input size   788MB Thrghput 56.38MB/s Time 00:00:15
➜  quickwit-indices cat mezmo/mezmo-use-stage-2023-01-20-ndjson/lines.njson | quickwit tool local-ingest --index mezmo
MAIN_BRUNCH
 Num docs   39859 Parse errs     0 PublSplits   0 Input size    48MB Thrghput 48.31MB/s Time 00:00:01
 Num docs   39859 Parse errs     0 PublSplits   0 Input size    48MB Thrghput 24.17MB/s Time 00:00:02
 Num docs   77716 Parse errs     0 PublSplits   0 Input size    95MB Thrghput 31.78MB/s Time 00:00:03
 Num docs  117262 Parse errs     0 PublSplits   0 Input size   144MB Thrghput 36.13MB/s Time 00:00:04
 Num docs  156652 Parse errs     0 PublSplits   0 Input size   192MB Thrghput 36.12MB/s Time 00:00:05
 Num docs  197282 Parse errs     0 PublSplits   0 Input size   242MB Thrghput 48.50MB/s Time 00:00:06
 Num docs  237082 Parse errs     0 PublSplits   0 Input size   291MB Thrghput 49.13MB/s Time 00:00:07
 Num docs  276808 Parse errs     0 PublSplits   0 Input size   341MB Thrghput 49.24MB/s Time 00:00:08
 Num docs  314865 Parse errs     0 PublSplits   0 Input size   388MB Thrghput 48.82MB/s Time 00:00:09
 Num docs  353884 Parse errs     0 PublSplits   0 Input size   437MB Thrghput 48.72MB/s Time 00:00:10
 Num docs  393704 Parse errs     0 PublSplits   0 Input size   485MB Thrghput 48.51MB/s Time 00:00:11
 Num docs  432803 Parse errs     0 PublSplits   0 Input size   534MB Thrghput 48.19MB/s Time 00:00:12
 Num docs  473526 Parse errs     0 PublSplits   0 Input size   583MB Thrghput 48.94MB/s Time 00:00:13
 Num docs  512875 Parse errs     0 PublSplits   0 Input size   633MB Thrghput 49.05MB/s Time 00:00:14
 Num docs  553857 Parse errs     0 PublSplits   0 Input size   682MB Thrghput 49.26MB/s Time 00:00:15

PSeitz avatar May 27 '24 05:05 PSeitz

MAIN_BRUNCH :)

fulmicoton avatar May 29 '24 03:05 fulmicoton