python-logging-loki icon indicating copy to clipboard operation
python-logging-loki copied to clipboard

Published 20 hours ago • verified publisher icon GreyZmeem

Improve performance with batching

Open theblazehen opened this issue 2 years ago • 3 comments

Currently it only uploads one log line with each HTTP request, which does not work well for high velocity logs.

The queued handler could potentially read all the available lines in the queue and upload everything outstanding at once as opposed to popping an item one at a time, and uploading it

theblazehen avatar Feb 20 '22 18:02 theblazehen

This would be nice feature, otherwise I'm little bit afraid to use it in production on high-load web servers. Did you come to any solution @theblazehen ?

tomashornak avatar Apr 08 '22 18:04 tomashornak

@tomashornak Afraid not, decided to just log to journald with https://pypi.org/project/systemd-logging/ and use promtail from there, but haven't tested it at scale yet

theblazehen avatar Apr 09 '22 07:04 theblazehen

@tomashornak Afraid not, decided to just log to journald with https://pypi.org/project/systemd-logging/ and use promtail from there, but haven't tested it at scale yet

How are you propagating unique_labels, tags etc in your solution?

nathabit avatar Apr 20 '22 11:04 nathabit