serilog-sinks-azureblobstorage icon indicating copy to clipboard operation
serilog-sinks-azureblobstorage copied to clipboard

Published 20 hours ago • verified publisher icon chriswill

WriteInBatches does not write all the data to the blob storage

Open Marien-OV opened this issue 2 years ago • 10 comments

Logging with the option writeInBatches on true does not write all the data to the blob storage. It looks that the PeriodicBatchingSink is not flushed or to early disposed. In the previous version (3.0.2) it works fine, but in 3.1.2 it doesn't. For now we use version 3.0.2. Hopefully you can solve this issue.

Marien-OV avatar Jun 14 '22 14:06 Marien-OV

Same problem here. I've also noticed w/ V3.1.2, that events are getting duplicated several times in the log as well. I went back to 3.0.2 and it all seems to work ok w/ no duplicated events. Hopefully, this all can be fixed in newer releases.

wildview avatar Jun 17 '22 17:06 wildview

I also noticed this problem

Jasonlhy avatar Jul 19 '22 09:07 Jasonlhy

Same problem here. Downgrading to 3.0.x works as before. For now, going with this :(

ZenwalkerD avatar Aug 24 '22 09:08 ZenwalkerD

I also have the same issue. I downgraded to 3.0.2 as suggested to get this working. This is a great feature to batch log to storage and I definitely want to take full advantage of this.

programmer-nate avatar Nov 16 '22 16:11 programmer-nate

Hello @chriswill, is this going to be fixed/worked on? We're running into the 50k block limit in the Azure Append blobs (so the file rolls over) and the batch writing would probably prevent/delay that. But if the batch writing functionality loses events that seems like a critical issue to address?

mikewop avatar Feb 06 '23 21:02 mikewop

@mikewop this will get worked on but I'm extremely busy with a project rn. If you think you can get a pull request put together for this, I'd be happy to review, include, and credit you.

chriswill avatar Feb 14 '23 23:02 chriswill

Any update on this? When can this be addressed?

ZenwalkerD avatar Mar 30 '23 17:03 ZenwalkerD

This seems to be important to many people, but no one seems to want to create a PR for it. I'm not uncaring, just over-scheduled and can use a hand on this.

chriswill avatar Apr 04 '23 06:04 chriswill

This seems to be important to many people, but no one seems to want to create a PR for it. I'm not uncaring, just over-scheduled and can use a hand on this.

Possible to give me a pointer as to what is the issue. I can look into providing a fix; if i can.

ZenwalkerD avatar May 16 '23 11:05 ZenwalkerD

I found two problems:

  1. In LoggerConfigurationAzureBlobStorageExtensions.cs: It reaches the queueLimit (10.000) that is configured in PeriodicBatchingSinkOptions. The default here is (100.000)
  2. In AzureBatchingBlobStorageSink.cs: for each logEvent a request done for the properties from the BlobClient. This takes too much time. This only needed when the blobname is changed.

Marien-OV avatar Jan 05 '24 12:01 Marien-OV

In the 4.0.0 version, the writeInBatches feature was totally written, to use the native Serilog support that shipped in their 4.0 release. Closing for now.

chriswill avatar Jul 17 '24 21:07 chriswill