Data Streams not working

[Schoof-T]

A few questions before you begin:

Is this an issue related to the Serilog core project or one of the sinks or community projects.
This issue list is intended for Serilog Elasticsearch Sink issues. If this issue relates to another sink or to the code project, please log on the related repository. Please use Gitter chat and Stack Overflow for discussions and questions.

Does this issue relate to a new feature or an existing bug?

• [x] Bug
• [ ] New Feature

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package. 8.4.1

What is the target framework and operating system? See target frameworks & net standard matrix.

• [x] .NET 6
• [ ] netCore 2.0
• [ ] netCore 1.0
• [ ] 4.7
• [ ] 4.6.x
• [ ] 4.5.x

Please describe the current behavior? According to the documentation I have to enable the following settings to enable logging to a data-stream.

TypeName = null,
IndexFormat = "logs-my-stream",
BatchAction = ElasticOpType.Create,

{
        "Name": "Elasticsearch",
        "Args": {
          "nodeUris": "http://vm-elasticdev01:9200",
          "indexFormat": "jdn-logging-stream-resttemplate",
          "templateName": "jdn-logging-template",
          "typeName": "",
          "batchAction": "Create"
        }
}

Note: using ElasticOpType.Create as batchAction does not work, that generates an error: System.ArgumentException: 'Requested value 'ElasticOpType.Create' was not found.'

I enabled them as following with my appsettings.json. But in kibana it does not show up under DataStreams.

I've also tried adding it codewise, but no luck.

        public static void Main(string[] args)
        {
            ConfigureLogging();

            try
            {
                Log.Information("Starting web host");
                CreateHostBuilder(args).Build().Run();
            }
            catch (Exception ex)
            {
                Log.Fatal(ex, "Host terminated unexpectedly");
            }
            finally
            {
                Log.CloseAndFlush();
            }
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
            .UseSerilog()
            //.UseSerilog((context, config) => config.ReadFrom.Configuration(context.Configuration))
            .ConfigureWebHostDefaults(webBuilder => webBuilder.UseStartup<Startup>());

        private static void ConfigureLogging()
        {
            var configuration = new ConfigurationBuilder()
                .AddJsonFile("appsettings.json")
                .Build();

            Log.Logger = new LoggerConfiguration()
                .WriteTo.Console()
                .WriteTo.Debug()
                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://vm-elasticdev01:9200"))
                {
                    BatchAction = ElasticOpType.Create,
                    TypeName = null,
                    IndexFormat = "jdn-logging-stream-resttemplate"
                    TemplateName = "jdn-logging-template",
                })
                .CreateLogger();

            //Log.Logger = new LoggerConfiguration()
            //    .ReadFrom.Configuration(configuration)
            //    .CreateLogger();
        }

Please describe the expected behavior? I would expect a DataStream to be created.

If the current behavior is a bug, please provide the steps to reproduce the issue and if possible a minimal demo of the problem

• Setup ElasticSearch, Kibana, SeriLog and SeriLog Sinks ElasticSearch as normal.
• Use the options provided in the documentation to enable creating a Data Stream
• Create a Data Stream index template (jdn-logging-template in my example)
• Log something
• No DataStream is created and Nothing is logged

[KiyoIchikawa]

Working from the example they have in the repo got us a working starting point: https://github.com/serilog-contrib/serilog-sinks-elasticsearch/tree/dev/sample/Serilog.Sinks.Elasticsearch.Sample

From there, the minimum working app settings for logging to ElasticSearch is below:

{
  "Serilog": {
    "Using": [
      "Serilog.Sinks.Elasticsearch"
    ],
    "WriteTo": [
      {
        "Name": "Elasticsearch",
        "Args": {
          "nodeUris": "http://xxx.xxx.xxx.xxx:9200",
          "batchAction": "Create",
          "autoRegisterTemplateVersion": "ESv7",
          "indexFormat": "serilog-dev",
          "customFormatter": "Serilog.Formatting.Elasticsearch.ElasticsearchJsonFormatter, Serilog.Formatting.Elasticsearch"
        }
      }
    ]
  }
}

The customFormatter is extra and not required. Something we discovered in working with this though, is that if your index format becomes dynamic in any way (using the date in the name serilog-{0:yyyy.MM.dd}) you will get a new datastream every day. I guess you could have the environment name injected in there somehow instead of the date, but I haven't gotten that far yet.