serilog-sinks-seq icon indicating copy to clipboard operation
serilog-sinks-seq copied to clipboard

Use size-based batch limiting

Open nblumhardt opened this issue 2 years ago • 12 comments

On the server-side, Seq imposes two size limits on incoming events:

  • The event size limit, which applies to the JSON-encoded byte size of individual events, and
  • The batch size limit, which applies to the total size (including newlines) of the CLEF-encoded POST body.

On the client side, this sink currently uses an event size limit, but batches are limited by count (batchPostingLimit) rather than byte size. This makes it difficult to reliably align the client and server settings, leading to potential batch loss when oversize batches are discarded, or poor batch utilization if the batch size has to be set to an excessively conservative low value.

A future version of this sink should replace batchPostingLimit with a size-based control.

The main caveat that will complicate this is the dependency on PeriodicBatchingSink, which implements the current count-based logic. The dependency on PeriodicBatchingSink will likely need to be broken and the functionality replaced in order to make the proposed change here.

nblumhardt avatar Mar 07 '22 05:03 nblumhardt