Logger compatibility with data streams

[morinow]

Elasticsearch.Extensions.Logging:

It seems to me, that unfortunately the Elasticsearch logger for .NET is not compatible with data streams.

Data streams require 'create' actions instead of 'index' to be used for bulk requests.

The ElasticsearchChannel class is currently determining the bulk action to be 'index' always, found in the method WriteBufferToStreamAsync:

private async Task WriteBufferToStreamAsync(List<TEvent> b, Stream stream, CancellationToken ctx)
{
	foreach (var @event in b)
	{
		if (@event == null) continue;
		var indexTime = Options.TimestampLookup?.Invoke(@event) ?? DateTimeOffset.Now;
		if (_options.IndexOffset.HasValue) indexTime = indexTime.ToOffset(_options.IndexOffset.Value);
		var index = string.Format(_options.Index, indexTime);

                // START PROBLEM
		var indexHeader = new { index = new { _index = index } };
                // END PROBLEM

		await JsonSerializer.SerializeAsync(stream, indexHeader, indexHeader.GetType(), ElasticsearchChannelStatics.SerializerOptions, ctx)
			.ConfigureAwait(false);
		await stream.WriteAsync(ElasticsearchChannelStatics.LineFeed, 0, 1, ctx).ConfigureAwait(false);
		if (Options.WriteEvent != null)
			await Options.WriteEvent(stream, ctx, @event).ConfigureAwait(false);
		else
		{
			await JsonSerializer.SerializeAsync(stream, @event, typeof(TEvent), ElasticsearchChannelStatics.SerializerOptions, ctx)
				.ConfigureAwait(false);
		}
		await stream.WriteAsync(ElasticsearchChannelStatics.LineFeed, 0, 1, ctx).ConfigureAwait(false);
	}
}

If my target index is a data stream, all log-event documents will be rejected by Elasticsearch.

It would make sense to have the desired operation type configurable for the logger.

[morinow]

For completeness, package version used was: 1.6.0-alpha1

[morinow]

Just discovered, that #110 addresses this issue by introducing different channel variations. Is there any realistic chance of this being realized in the near future? Tbh this seems like a huge deal breaker, since many will probably use data streams for log scenarios.

[Mpdreamz]

@morinow Thanks for raising this issue and apologies for the radio silence on our end.

#110 is now merged and we are gearing up to release an 8.4.0, this will include a rewritten Elasticsearch.Extensions.Logging that takes advantage of the new DataStreamChannel.

Closing this issue ahead of the release (Date still TBD).