Integrate `Elasticsearch.CommonSchema.Serilog` text formatter

[nenadvicentic]

A few questions before you begin:

Is this an issue related to the Serilog core project or one of the sinks or community projects.
This issue list is intended for Serilog Elasticsearch Sink issues. If this issue relates to another sink or to the code project, please log on the related repository. Please use Gitter chat and Stack Overflow for discussions and questions.

Does this issue relate to a new feature or an existing bug?

• [ ] Bug
• [x] New Feature

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package. 9.0.0-alpha-*

What is the target framework and operating system? See target frameworks & net standard matrix.

• [x] net 7.0
• [x] net 6.0
• [x] 4.8

As a part of integration effort between .NET and Elasticsearch, Elastic team provides Elastic Common Schema .NET repository which includes two sub-projects:

• Elastic Common Schema .NET which contains a full C# representation of Elastic Common Schema (ECS) and esc-* index templates.
• Elastic Common Schema Serilog Text Formatter which provides Serilog.Formatting.ITextFormatter implementation in the form of Elastic.CommonSchema.Serilog.EcsTextFormatter, for easy integration with Serilog:

  var logger = new LoggerConfiguration()
              .WriteTo.Console(new EcsTextFormatter())
              .CreateLogger();
  

The issue is a follow-up to #254 and #227, containing more up-to-date information and first look at the integration.

Simple experiment with naïve Serilog.Sinks.Elasticsearch integration, where EcsTextFormatter simply replaces current default formater ElasticsearchJsonFormatter:

public static ITextFormatter CreateDefaultFormatter(ElasticsearchSinkOptions options)
{
    //return new ElasticsearchJsonFormatter(
    //    formatProvider: options.FormatProvider,
    //    closingDelimiter: string.Empty,
    //    serializer: options.Serializer != null ? new SerializerAdapter(options.Serializer) : null,
    //    inlineFields: options.InlineFields,
    //    formatStackTraceAsArray: options.FormatStackTraceAsArray
    //);

    return new EcsTextFormatter();
}

...produces following output in Elasticsearch/Kibana v8.6.0:

Open questions would be:

• What would be minimal acceptable implementation (e.g. default EcsTextFormatter settings work just fine with Elasticsearch, using dynamic fields mapping, without any template registration)? Full implementation, with parity of features is something that would probably take a lot of rewrite effort and testing.
• What to do with existing formatters? Should they be supported side-by-side (at least for the moment)?

[nenadvicentic]

Currently, using Serilog.Sinks.Elasticsearch v9.0.0, it is possible to integrate Elastic.CommonSchema.Serilog.EcsTextFormatter using the customFormatter options, as shown in the extract of appsettings.json bellow:

{
  // ....
  "Serilog": {
    "Using": [ "Serilog.Sinks.Elasticsearch" ],
    "MinimumLevel": "Information",
    "WriteTo": [
      {
        "Name": "Elasticsearch",
        "Args": {
          "nodeUris": "http://localhost:9200",
          "customFormatter": "Elastic.CommonSchema.Serilog.EcsTextFormatter, Elastic.CommonSchema.Serilog"
        }
      }
    ],
    "Enrich": [ "FromLogContext", "WithMachineName" ],
    "Properties": {
      "Application": "My app"
    }
  },
  // ...
}

However, it turned out that current version of Elastic.CommonSchema.Serilog.EcsTextFormatter from Elastic.CommonSchema.Serilog NuGet package v1.5.3 only works with Elasticsearch server v8.x! It breaks logging when pointed to Elasticsearch server versions lower than 8. With it's default formatter Serilog.Sinks.Elasticsearch v9.0.0 works against Elasticsearch server major versions 6, 7 and 8.