Andriy Redko

> @jainankitk @msfroh @reta would love some feedback Apologies @harshavamsi , will take a look once the release is cleared out (later this week), thank you

There is new experimental API introduced in `2.15.0` which is not yet documented nor finalized: ``` POST /_bulk/stream PUT /_bulk/stream POST /{index}/_bulk/stream PUT /{index}/_bulk/stream ``` The new API is using...

### Option `#1`: Keep using NDJSON for streaming ingestion The current `_bulk` APIs use NDJSON (Newline delimited JSON, https://github.com/ndjson/ndjson-spec) which fits perfectly to the streaming use cases: each chunk sent...

@andrross @dblock @msfroh would appreciate early feedback if possible, working on some POCs at the moment to capture the operational metrics.

> @reta My inclination here is that since this is a new API that is specifically targeting high-throughput use cases then we should explore option 2. Thanks @andrross , I...

> @reta I'm really curious if the protobuf `Any` type is feasible here. Have you actually used it in practice at all? I was thinking we might have to fallback...

To have change delivered incrementally, focusing this issue on HTTP/NDJSON API flavour (Option `#1`) and spinning off Protobuf API flavour into https://github.com/opensearch-project/OpenSearch/issues/15447 for future release (Option `#2`).

Thanks a lot, @finnegancarroll , I am a bit surprised (since "in general" the specialized impl is expected to be a bit faster than generalized one). I am eager to...

Thanks @finnegancarroll > `SearchHitsProtobufBenchmark.writeToNativeBench` spends almost 40% of its time in `StreamOutput::writeGenericValue` but I don't see the same kind of time spent writing to stream in the protobuf implementation. Very...

Thanks @finnegancarroll , my apologies for the delay, just completed may part. I didn't spend too much time on native serialization but it looks like (with the benchmarks) `StreamOutput::getGenericType` (that...