quickwit
quickwit copied to clipboard
use serde_json_borrow to parse JSON
- update tantivy
- use serde_json_borrow instead of serde_json::Value
➜ quickwit-indices cat mezmo/mezmo-use-stage-2023-01-20-ndjson/lines.njson | quickwit tool local-ingest --index mezmo
serde_json_borrow + CompactDoc
Num docs 46475 Parse errs 0 PublSplits 0 Input size 56MB Thrghput 28.42MB/s Time 00:00:02
Num docs 94511 Parse errs 0 PublSplits 0 Input size 115MB Thrghput 38.54MB/s Time 00:00:03
Num docs 138769 Parse errs 0 PublSplits 0 Input size 170MB Thrghput 42.67MB/s Time 00:00:04
Num docs 184161 Parse errs 0 PublSplits 0 Input size 225MB Thrghput 45.15MB/s Time 00:00:05
Num docs 227811 Parse errs 0 PublSplits 0 Input size 280MB Thrghput 56.02MB/s Time 00:00:06
Num docs 273217 Parse errs 0 PublSplits 0 Input size 337MB Thrghput 55.50MB/s Time 00:00:07
Num docs 320748 Parse errs 0 PublSplits 0 Input size 395MB Thrghput 56.17MB/s Time 00:00:08
Num docs 366026 Parse errs 0 PublSplits 0 Input size 451MB Thrghput 56.45MB/s Time 00:00:09
Num docs 410292 Parse errs 0 PublSplits 0 Input size 506MB Thrghput 56.45MB/s Time 00:00:10
Num docs 456037 Parse errs 0 PublSplits 0 Input size 562MB Thrghput 56.26MB/s Time 00:00:11
Num docs 501879 Parse errs 0 PublSplits 0 Input size 619MB Thrghput 56.09MB/s Time 00:00:12
Num docs 549379 Parse errs 0 PublSplits 0 Input size 677MB Thrghput 56.47MB/s Time 00:00:13
Num docs 594425 Parse errs 0 PublSplits 0 Input size 732MB Thrghput 56.45MB/s Time 00:00:14
Num docs 638732 Parse errs 0 PublSplits 0 Input size 788MB Thrghput 56.38MB/s Time 00:00:15
➜ quickwit-indices cat mezmo/mezmo-use-stage-2023-01-20-ndjson/lines.njson | quickwit tool local-ingest --index mezmo
MAIN_BRUNCH
Num docs 39859 Parse errs 0 PublSplits 0 Input size 48MB Thrghput 48.31MB/s Time 00:00:01
Num docs 39859 Parse errs 0 PublSplits 0 Input size 48MB Thrghput 24.17MB/s Time 00:00:02
Num docs 77716 Parse errs 0 PublSplits 0 Input size 95MB Thrghput 31.78MB/s Time 00:00:03
Num docs 117262 Parse errs 0 PublSplits 0 Input size 144MB Thrghput 36.13MB/s Time 00:00:04
Num docs 156652 Parse errs 0 PublSplits 0 Input size 192MB Thrghput 36.12MB/s Time 00:00:05
Num docs 197282 Parse errs 0 PublSplits 0 Input size 242MB Thrghput 48.50MB/s Time 00:00:06
Num docs 237082 Parse errs 0 PublSplits 0 Input size 291MB Thrghput 49.13MB/s Time 00:00:07
Num docs 276808 Parse errs 0 PublSplits 0 Input size 341MB Thrghput 49.24MB/s Time 00:00:08
Num docs 314865 Parse errs 0 PublSplits 0 Input size 388MB Thrghput 48.82MB/s Time 00:00:09
Num docs 353884 Parse errs 0 PublSplits 0 Input size 437MB Thrghput 48.72MB/s Time 00:00:10
Num docs 393704 Parse errs 0 PublSplits 0 Input size 485MB Thrghput 48.51MB/s Time 00:00:11
Num docs 432803 Parse errs 0 PublSplits 0 Input size 534MB Thrghput 48.19MB/s Time 00:00:12
Num docs 473526 Parse errs 0 PublSplits 0 Input size 583MB Thrghput 48.94MB/s Time 00:00:13
Num docs 512875 Parse errs 0 PublSplits 0 Input size 633MB Thrghput 49.05MB/s Time 00:00:14
Num docs 553857 Parse errs 0 PublSplits 0 Input size 682MB Thrghput 49.26MB/s Time 00:00:15
MAIN_BRUNCH :)