Level property keeps getting renamed and added to the LokiLogEvent if the grafana server is down

[mcauzzi]

Which version of Serilog.Sinks.Grafana.Loki are you using?

v8.3.0

Which version of .NET are you using?

net6.0

Describe the bug

If a socketException is thrown, the field _waitingBatch in the class LokiSink is not cleared (while it is cleared if an error StatusCode is returned). Next time the timer is fired, the same logs get sent out, the level property gets renamed to "level" and a new "level" property is added. Every following tick a new level property gets added and and the old one gets renamed with an ever increasing number of pre-pended "".

First timer tick: The first level gets added

Second timer tick: The second level property gets added

Fourth tick and beyond: A new level property keeps getting added

To Reproduce

Connect to an address with no Loki Instance attached to it

Expected behavior

The number of "level" properties should not grow.

Log/SelfLog output or exception with stacktrace

No response

Application or code sample, which could be used to reproduce a bug

No response

Additional context

Our grafana instance went down for the weekend and the memory consumption went from 400MB~ to 3GB~ for each instance of our deployed application. Every single logevent in the memory dump had hundreds of level properties with an increasing number of underscores.

A screenshot of a log property extracted from our app:

We fixed the problem by adding a custom property renamer:

public class GrafanaPropertiesRenamingStrategy:IReservedPropertyRenamingStrategy
{
    public string Rename(string originalName) => originalName=="level"?originalName:$"_{originalName}";
}

I have read the documentation

• [X] I have read Common Problems section in project wiki