Duplicated logs when using buffer

[gjfonte]

trafficstars

What version of Serilog.Sinks.Elasticsearch is affected? 8.0.0

What is the target framework and operating system?

• [x] 4.6.2

Current behavior? Logs sent to Elasticsearch cluster are duplicated (triplicated or more).

Expected behavior? Should be saved one log only per entry. Same buffer settings are working fine with version 7.1.0

Steps to reproduce the issue and if possible a minimal demo of the problem

This is my current code:

private static ILogger GetLogger()
{
    string indexName = "buffertest";

    var loggerConfig = new LoggerConfiguration()
        .MinimumLevel.Debug()
        .WriteTo.Console();
        
    var esSink = new ElasticsearchSinkOptions(new Uri("https://myelasticdomain.com:9200"))
    {
        AutoRegisterTemplate = true,
        AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
        TemplateName = indexName,
        IndexFormat = indexName + "-{0:yyyy.MM.dd}",
        ModifyConnectionSettings = c => c.BasicAuthentication("myuser", "****"),
        CustomFormatter = new ExceptionAsObjectJsonFormatter(renderMessage: true)
    };

    // buffer
    esSink.BufferBaseFilename = $@"C:\logs\ElasticSeriLog\buffer-{indexName}";
    esSink.SingleEventSizePostingLimit = null;
    esSink.BufferFileSizeLimitBytes = 1000 * 1000 * 100; // 100 MB
    esSink.BufferFileCountLimit = 50;
    
    loggerConfig.WriteTo.Elasticsearch(esSink);

    return loggerConfig.CreateLogger();
}
    
static void Main(string[] args)
{
    Log.Logger = GetLogger();

    for (int i = 0; i < 500; i++)
    {
        var lineNumer = i + 1;
        var msgToLog = GetTextToLog();
        Log.Information("{Line}{Message}", lineNumer, msgToLog);
    }

    Console.WriteLine("Press any key to continue...");
    Console.ReadKey();
    
    Console.WriteLine("Flushing...");
    Log.CloseAndFlush();
}`

[ffMathy]

I have this problem too.

@gjfonte is there a way to enable a workaround in the meanwhile?

[sharpsalat]

Have this problem too, only with buffer.

[sharpsalat]

i've installed this think from github repository and from nuget 8.1.0-alpha0002, and didn't notice this problem. i guess we need to use until it'll fixed in next stable release.

[AbrahamLopez10]

I have this problem as well, with Elasticsearch 7.4, Serilog.Sinks.Elasticsearch 8.0.0, Serilog 2.8.0, and .NET Core 2.2.

I also noticed that often the logs will not get sent to ES for a while, and when they eventually get sent, each one will appear repeated several times in ES.

[sharpsalat]

I have this problem as well, with Elasticsearch 7.4, Serilog.Sinks.Elasticsearch 8.0.0, Serilog 2.8.0, and .NET Core 2.2.

I also noticed that often the logs will not get sent to ES for a while, and when they eventually get sent, each one will appear repeated several times in ES.

https://www.nuget.org/packages/Serilog.Sinks.ElasticSearch/8.1.0-alpha0002 So i guess we should wait for realease with this fix

[jerpic]

Hello, I upgraded to the lastest version 8.1.0-alpha006, it seems the issue is still present. Anybody knows when this issue will be fix? The number of items in the buffer file is correct, however Elastic receives the items several times for some of them. Regards,

[IngemarNilsson]

Same problem here. Logs not sent and/or duplicates in Elastic. Downgraded to version 7.1.0

[ruicarvalhoPT]

Yap.. Same problem. Also downgraded to version 7.1.0