prometheus-nginxlog-exporter icon indicating copy to clipboard operation
prometheus-nginxlog-exporter copied to clipboard

Published 20 hours ago • verified publisher icon martin-helmich

Improve Performance For "MaxLineSize" and Concurrency of ParseString(line)

Open lhanjian opened this issue 3 years ago • 2 comments

https://github.com/martin-helmich/prometheus-nginxlog-exporter/blob/f09c9dc27715b139c4ca97917fbd736905649e20/tail/tailer.go#L44

add MaxLineSize: 65534 to improve Throughput of reading file?

https://github.com/martin-helmich/prometheus-nginxlog-exporter/blob/f09c9dc27715b139c4ca97917fbd736905649e20/main.go#L372

to Concurrent Execute ParseString(line) in this loop?

over 25000 QPS access.log, exporter unable to update metrics accurately.

Tks.

lhanjian avatar Nov 27 '21 13:11 lhanjian

25k qps is a respectable workload 😳 -- admittedly, this is the heaviest workload that I've seen this exporter been subjected to. I'd be happy to include optimizations for such heavy workloads, however I can't make any promises as to when I'll actually find the time to get to that. In the meantime, I'm happy to accept any PRs in this matter. 🙂

martin-helmich avatar Dec 13 '21 19:12 martin-helmich

Do you have a rule of thumb for the max qps that you know the exporter can handle without pegging the one core that it uses and falling behind on log processing?

gburton1 avatar Jan 03 '22 19:01 gburton1