Serilog.Sinks not working after upgrading to Elasticsearch 8.1.3

[Wmarie]

trafficstars

Does this issue relate to a new feature or an existing bug?

• [x] Bug
• [ ] New Feature

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package. 8.4.1 and 9.0.0-beta4

What is the target framework and operating system? See target frameworks & net standard matrix.

• [x] .Net Core 3.1
• [x] .Net Standard 2.0
• [x] .Net Framework 4.7.2 The main runtime is .Net Core 3.1 but I also have a .Net Framework 4.7.2 project where i would like it to work.

Please describe the current behavior? After updating to Elasticsearch 8.1.3 I am unable to write logs using serilog sinks.

The logs are written correctly to the bufferBaseFile and I can successfully Post an entry manually to a Elasticsearch 8.1.3 using Dev Console in Kibana without setting any component/index templates.

From Serilog Selflog i get the error:

Exception while emitting periodic batch from Serilog.Sinks.Elasticsearch.Durable.ElasticsearchLogClient: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 400 from: POST /_bulk
   at Elasticsearch.Net.Transport`1.HandleElasticsearchClientException(RequestData data, Exception clientException, IElasticsearchResponse response)
   at Elasticsearch.Net.Transport`1.FinalizeResponse[TResponse](RequestData requestData, IRequestPipeline pipeline, List`1 seenExceptions, TResponse response)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   at Serilog.Sinks.Elasticsearch.Durable.ElasticsearchLogClient.SendPayloadAsync(List`1 payload, Boolean first) 

I have tried with the 8..4.1 version, and have added the writeTo settings that should help with backwards compatability. Serilog creates a legacy index template that matches my provided IndexPattern.

I haven't been able to update to Serilog.Sinks.Elasticsearch version 9.0.0-beta4, as it requires a Serilog.Formatting.Elasticsearch which does not exist: Serilog.Sinks.Elasticsearch 9.0.0-beta4 -> Serilog.Formatting.Elasticsearch (>= 9.0.0-beta4) Please describe the expected behavior? I am expecting to be able to see the logs that ar written to the BufferFile, being written to Elasticsearch to an Index matching my IndexPattern.

If the current behavior is a bug, please provide the steps to reproduce the issue and if possible a minimal demo of the problem

"WriteTo": [
      {
        "Name": "Elasticsearch",
        "Args": {
          "nodeUris": "{elastic cloud URL}",
          "indexFormat": "st-log-event-{0:yyyy.MM.dd}",
          "bufferBaseFilename": "C:\\Temp\\ElasticsearchEventBuffer_Web",
          "connectionGlobalHeaders": "Authorization=Basic xxxxx=",
          "autoRegisterTemplate": true,
          "autoRegisterTemplateVersion": "ESv7",
          "typeName": null,
          "batchAction": "Create",
          "detectElasticsearchVersion": true,
          "registerTemplateFailure": "IndexAnyway"
        }
      },
      {
        "Name": "Console"
      }
    ]

var loggerConfiguration = new LoggerConfiguration()
                .ReadFrom.Configuration(configuration, "SerilogEvent")
.Enrich.FromLogContext()
.CreateLogger();

[shokri-navid]

I faced same Issue today and It works only with in-code configuration and it seems appsetting.json cannot set TypeName to null value. it should be documented in your readme file to notice the users at least

[Wmarie]

Thank you @navid-dada! Made it work by moving all my settings to in-code configuration and including the TypeName = null, with version 8.4.1.

Would still be nice to be able to use appsettings.json though, but this works for now :)

[copernicus365]

Yes, this fixed it for me as well, thanks! To be clear,

0. going against elasticsearch version 8.2.0
1. while using Serilog.Formatting.Elasticsearch version 9.0.0-beta7 (I upgraded to highest available, maybe not needed to even be version 9, don't know)
2. Serilog.Sinks.Elasticsearch version 9.0.0-beta7
3. For ElasticsearchSinkOptions set TypeName = null,

The last setting fixed the problem I was getting, which you could see when setting within ElasticsearchSinkOptions.ModifyConnectionSettings: config.OnRequestCompleted(d => $"es-req: {d.DebugInformation}".Print())

The above let it be seen the problem was this (from the printed out DebugInformation, there was a illegal_argument_exception being returned from elasticsearch: Action/metadata line [1] contains an unknown parameter [_type]. Full response:

• [1] BadResponse: Node: https://xyz.elastic-cloud.com:1234/ Took: 00:00:00.0447236

OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 400 from: POST /_bulk. ServerError: Type: illegal_argument_exception Reason: "Action/metadata line [1] contains an unknown parameter [_type]"

[bandersen22000]

Any update on when the fix will be added to a package so we can keep the settings in appsettings.json ?

[YoussefWaelMohamedLotfy]

Read this article. It helped me to get the Sink working. Write to ElasticSearch 8 with Serilog in .NET Core - Article

Take note of the ElasticsearchOptions URI as it should contain the username and password, unlike in v7. I hope this helps.

[Arthur-Neto]

Read this article. It helped me to get the Sink working. Write to ElasticSearch 8 with Serilog in .NET Core - Article

Take note of the ElasticsearchOptions URI as it should contain the username and password, unlike in v7. I hope this helps.

Shouldn't that work with the property "connectionGlobalHeaders" too? Because it's not.

[smg-bg]

I can create a PR about Elasticsearch 8.x functionality. There are just a couple of lines, but let’s see how testing will go….

[mdg215199]

Any update on this?

[jonasmidstrup]

Making the sink work with Elasticsearch 8.x is also captured here: #460