Proposal: index arrays instead of newline delimited "streams"

[fergiemcdowall]

Personally although I like the promise of streams,I think that they are becoming an increasingly esoteric way of reading large blobs of data, and are creating a barrier to people who just want to get going with Norch.

Whould anybody be unhappy if we changed the /add endpoint to accept arrays instead of streams?

[eklem]

I love the indexing part of the stream API on search-index, but since we're talking about the server-side here, I don't think it's a problem.

But maybe be able to add streams as well, even if through some tools. Today search-index has add and concurrentAdd (for arrays). You're thinking of swapping to the concurrentAdd? The memory issues of indexing went away with streams, so for large data-sets we should maybe keep a way to do this for search-index, while the norch-endpoints could be the user friendlier version?

[fergiemcdowall]

@eklem I feel like it would be better if Norch's add accepted standard JSON.parse-able objects as default with the option to specify more advanced inputs such as streams as an add-on, or preprocessing stage

[eklem]

Yes, sounds good to have add as the standard choice with easy-to-use arrays as input. Also have some dirty tricks to parse streams in the browser when querying a norch-instance. Could be nice to get rid of those too =)

[fergiemcdowall]

Upcoming version is de-streamified