Sink AWS ElasticSearch - Elasticsearch.Net.Utf8Json.JsonParsingException

[nicolastakashi]

trafficstars

A few questions before you begin:

Is this an issue related to the Serilog core project or one of the sinks or community projects.
This issue list is intended for Serilog Elasticsearch Sink issues. If this issue relates to another sink or to the code project, please log on to the related repository. Please use Gitter chat and Stack Overflow for discussions and questions.

Does this issue relate to a new feature or an existing bug?

• [x] Bug
• [ ] New Feature

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package.

• 8.1.0

What is the target framework and operating system? See target frameworks & net standard matrix.

• [x] netcore 3.1
• [ ] netCore 2.0
• [ ] netCore 1.0
• [ ] 4.7
• [ ] 4.6.x
• [ ] 4.5.x

Please describe the current behavior?

When trying to log into AWS ElasticSearch, the below exception is thrown:

Elasticsearch.Net.Utf8Json.JsonParsingException: expected:'{', actual:'U', at offset:0
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadIsBeginObjectWithVerify()
   at Elasticsearch.Net.Utf8Json.Formatters.DictionaryFormatterBase`5.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Elasticsearch.Net.DynamicDictionaryFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.LowLevelRequestResponseSerializer.Deserialize[T](Stream stream)
   at Elasticsearch.Net.ResponseBuilder.SetSpecialTypes[TResponse](Byte[] bytes, IMemoryStreamFactory memoryStreamFactory, TResponse& cs)
   at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
   at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)
   at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)

I'm assuming the root cause of the issue is the sink, which is causing ElasticSearch.Net to throw the exception, but I could be mistaken.

I've configured CreateHostBuild method as follows:

var elasticSearchSinkOptions = new ElasticsearchSinkOptions(new Uri(configuration.GetConnectionString("ElasticSearch")))
            {
                IndexFormat = "api-logs-{0:yyyy.MM.dd}",
                AutoRegisterTemplate = true,
                CustomFormatter = new ExceptionAsObjectJsonFormatter(renderMessage: true),
                EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog | EmitEventFailureHandling.WriteToFailureSink | EmitEventFailureHandling.RaiseCallback,
                FailureCallback = @event => { Console.WriteLine(@event); },
                ModifyConnectionSettings = conn => conn
                    .RequestTimeout(TimeSpan.FromMinutes(1))
                    .BasicAuthentication(configuration["Elastic:User"], configuration["Elastic:Password"])
                    .DisableAutomaticProxyDetection()
                    .DisableDirectStreaming(),
            };

            
            Log.Logger = new LoggerConfiguration()
                .MinimumLevel.Information()
                .Enrich.FromLogContext()
                .Enrich.WithExceptionDetails()
                .WriteTo.Elasticsearch(elasticSearchSinkOptions)
                .CreateLogger();
            
            return services;

In the controller, I've tried sending just a plain string with an exception, with the same above exception:

try
{
    //DoSomethingThatThrowsAnException()
}
catch(Exception ex)
{
   Log.Error(ex, "Controller.Exceptioon")
}

Please describe the expected behavior?

If the current behavior is a bug, please provide the steps to reproduce the issue and if possible a minimal demo of the problem

No exception and logs are sent to ES.

[mivano]

Looking at the call stack, it is trying to deserialize something from ES, but is unable to do so. I m not sure how this relates to the sink itself. Can you capture what is being returned? With Fiddler for example? Or see on the ES server in the logs what is happening. Might be that ES client is hitting something else than the ES server and is trying to parse that response.