simd-lite
Add support for newlin-delimited JSON (NDJSON)
Summary
#188 as an exercise showed that the feature to work with newline-delimited JSON (NDJSON) is not implemented in this crate.
Why
This feature is helpful when you have large number of records but each of those records are small JSON objects per line. This is often the case with large JSON files and looping over them and calling simd-json on each line is not going to help. This is added by @Licenser in this comment:
Ja the lines are fairly short too the advantages are a lot smaller (sometimes detrimental) as there is an initial cost to pay for filling the registers, doing multiple runs etc. can overshadow the performance gain for very small payloads.
@Licenser also adds
NDJSON would be incredibly cool (especially if we manage to realize in a streaming fashion / as an iterator)
What
Upstream simdjson has this feature called parse_many. Porting that to this crate is the first step.
!!!NEEDS MORE DETAILS!!!
Just sketching something here. An API that would be really nice would be something like (non valid rust syntax but pseude code!
fn iter_lines(r: Read) -> Iter<simd-json::DeserilizeableType>;
for items in iter_lines(file) {
do_stuff(item)
}
