Segmentation fault Serilog.Sinks.Elasticsearch version 9.0.0 with Elasticsearch version 7.17 on macOS

[maxim-kozlov]

trafficstars

Does this issue relate to a new feature or an existing bug?

• [x] Bug
• [ ] New Feature

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package.

• Serilog.Sinks.Elasticsearch 8.4.1
• Serilog.Formatting.Elasticsearch 8.4.1
• Serilog.Sinks.Elasticsearch 9.0.0
• Serilog.Formatting.Elasticsearch 9.0.0

Others NuGet packages.

• Serilog 2.12.0
• Serilog.AspNetCore 6.1.0
• Serilog.Settings.Configuration 3.4.0

What is the target framework and operating system? See target frameworks & net standard matrix.

• [x] .net 6

Please describe the current behavior? When using Serilog.Sinks.Elasticsearch 9.0.0 on macOS:

dotnet TestService.dll
[1]    15227 segmentation fault  dotnet TestService.dll

When using Serilog.Sinks.Elasticsearch 8.4.1 on macOS:

dotnet TestService.dll
[01:05:49 INF] Test <s:>

If the current behavior is a bug, please provide the steps to reproduce the issue and if possible a minimal demo of the problem

using Serilog;

try
{
    var conf = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json")
        .Build();

    Log.Logger = new LoggerConfiguration()
        .ReadFrom.Configuration(conf)
        .CreateBootstrapLogger();

    Log.Logger.Information("Test");
}
catch (Exception e)
{
    Log.Fatal(e, "Stopped program because of exception!");
}
finally
{
    await Log.CloseAndFlushAsync();
}

appsettings.json

{
  "Serilog": {
    "WriteTo": [
      {
        "Name": "Console",
        "Args": {
          "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} <s:{SourceContext}>"
        }
      },
      {
        "Name": "Elasticsearch",
        "Args": {
          "nodeUris": "http://localhost:9200/;",
          "indexFormat": "Test-Server-{0:yyyy.MM}",
          "inlineFields": true,
          "batchPostingLimit": 30,
          "restrictedToMinimumLevel": "Debug",
          "bufferFileSizeLimitBytes": 5242880,
          "bufferLogShippingInterval": 5000,
          "bufferRetainedInvalidPayloadsLimitBytes": 5000,
          "bufferFileCountLimit": 31,
          "connectionTimeout": 5,
          "emitEventFailure": "WriteToSelfLog",
          "queueSizeLimit": "100000",
          "autoRegisterTemplate": true,
          "autoRegisterTemplateVersion": "ESv7",
          "overwriteTemplate": false,
          "registerTemplateFailure": "IndexAnyway",
          "deadLetterIndexName": "deadletter-{0:yyyy.MM}"
        }
      }
    ]
  }
}

docker-compose.yaml

version: '3.7'

services:
  elasticsearch:
    image: elasticsearch:7.17.0
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      ES_JAVA_OPTS: "-Xmx2048m -Xms2048m"
      discovery.type: single-node

[nenadvicentic]

@maxim-kozlov Can you please try few additional things and look if there are additional logs on the machine, related to the "Segmantation fault" error?

• Is your program running correctly only with Serilog (remove Elasticsearch section in appsettings.json and remove reference to the Serilog.Sinks.Elasticserach? We just want to make sure it is not a problem with .NET runtime or something related to build/platform configuration.
• Is there any detailed exception log? Perhaps on the machine level. I run into this suggestion on StackOverflow, the paths where you might find log files, but it could be elsewhere too.
• Which exact version of Serilog.Sinks.Elasticsearch and Elasticsearch.NET is your program using? You listed both 8.4.1 and 9.0.0. in the details. If this is correct, are both version throwing the same error?

[gonace]

We're seeing the same problem but only for ASP.NET applications, console applications work just fine.

Versions tested:

• 9.0.0
• 9.0.1

I attached the crash report from one of my colleagues M1 MacBook Pro.

Crash-Report-20230607100100.txt

[dvakor]

Also catch seg.fault with [Serilog.Sinks.Elasticsearch 9.0.3] on M1 Pro (https://www.nuget.org/packages/Serilog.Sinks.Elasticsearch/9.0.3)

If remove ElasticSearch section from config its works fine.

[dvakor]

I make some researches and found, that its runtime bug >.<

When we load sink from appsettings its call specific extension method via reflection.

In lasts updates new parameter was added: detectElasticsearchVersion in LoggerConfigurationElasticsearchExtensions.Elasticsearch After this its start crashing.

Idk why, but only 36 parameters is a max number that reflection call can handle in this combinations of parameters types, after adding more its crashed. I create extension method with 40 params of string/int and its works :)

Its only specific for osx arm environment.

Minimal example for reproduce:

GetImplicitValueForNotSpecifiedKey taken from serilog sources and a little bit simplified

var methodInfo = typeof(LoggerConfigurationElasticsearchExtensions)
    .GetMethod(nameof(LoggerConfigurationElasticsearchExtensions.Elasticsearch));

var parameters = methodInfo.GetParameters()
    .Select(p => GetImplicitValueForNotSpecifiedKey(p, methodInfo)).ToArray();

Console.WriteLine("Before call");

methodInfo.Invoke((object)null, parameters); // Seg. fault here

Console.WriteLine("After call");

static object? GetImplicitValueForNotSpecifiedKey(
    ParameterInfo parameter,
    MethodInfo methodToInvoke)
{
    if (!(parameter.ParameterType == typeof (IConfiguration)))
        return null;
    if (parameter.HasDefaultValue)
    {
        if (parameter.DefaultValue.GetType() == typeof(DBNull))
        {
            return null;
        }
        return parameter.DefaultValue;
    }
    DefaultInterpolatedStringHandler interpolatedStringHandler = new DefaultInterpolatedStringHandler(81, 1);
    interpolatedStringHandler.AppendLiteral("This is not supported when only a `IConfigSection` has been provided. (method '");
    interpolatedStringHandler.AppendFormatted<MethodInfo>(methodToInvoke);
    interpolatedStringHandler.AppendLiteral("')");
    throw new InvalidOperationException("Trying to invoke a configuration method accepting a `IConfiguration` argument. " + interpolatedStringHandler.ToStringAndClear());
}

public static LoggerConfiguration Elasticsearch(
            this LoggerSinkConfiguration loggerSinkConfiguration,
            string nodeUris,
            string indexFormat = null,
            string templateName = null,
            string typeName = null,
            int batchPostingLimit = 50,
            int period = 2,
            bool inlineFields = false,
            LogEventLevel restrictedToMinimumLevel = LevelAlias.Minimum,
            string bufferBaseFilename = null,
            long? bufferFileSizeLimitBytes = null,
            long bufferLogShippingInterval = 5000,
            string connectionGlobalHeaders = null,
            LoggingLevelSwitch levelSwitch = null,
            int connectionTimeout = 5,
            EmitEventFailureHandling emitEventFailure = EmitEventFailureHandling.WriteToSelfLog,
            int queueSizeLimit = 100000,
            string pipelineName = null,
            bool autoRegisterTemplate = false,
            AutoRegisterTemplateVersion? autoRegisterTemplateVersion = null,
            bool overwriteTemplate = false,
            RegisterTemplateRecovery registerTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
            string deadLetterIndexName = null,
            int? numberOfShards = null,
            int? numberOfReplicas = null,
            IFormatProvider formatProvider = null,
            IConnection connection = null,
            IElasticsearchSerializer serializer = null,
            IConnectionPool connectionPool = null,
            ITextFormatter customFormatter = null,
            ITextFormatter customDurableFormatter = null,
            ILogEventSink failureSink = null,
            long? singleEventSizePostingLimit = null,
            int? bufferFileCountLimit = null,
            Dictionary<string,string> templateCustomSettings = null,
            ElasticOpType batchAction = ElasticOpType.Index,
            bool detectElasticsearchVersion = false
            )
        {
            return null;
        }

Can we reduce params count somehow?