Bryan Boreham

Why do we have two configmaps for promtail that look very similar?

Why is it running 3 promtails? This seems unnecessary.

The DaemonSet only runs on one node, due to https://github.com/prometheus/test-infra/blob/master/prombench/manifests/cluster-infra/6d_promtail_daemon_set.yaml#L124 It’s unnecessary to have all this complication; I’ll fix it.

Cleanup is in #749. Currently running that config [here](http://prombench.prometheus.io/grafana/explore?schemaVersion=1&panes=%7B%225f8%22:%7B%22datasource%22:%22PEEA25C9CAAE5E714%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22PEEA25C9CAAE5E714%22%7D,%22editorMode%22:%22code%22,%22expr%22:%22%7Bcontainer_name%21%3D%5C%22%5C%22%7D%20%22,%22queryType%22:%22range%22%7D%5D,%22range%22:%7B%22from%22:%22now-5m%22,%22to%22:%22now%22%7D,%22panelsState%22:%7B%22logs%22:%7B%22columns%22:%7B%220%22:%22Time%22,%221%22:%22Line%22,%222%22:%22container_name%22%7D,%22visualisationType%22:%22logs%22,%22labelFieldName%22:%22labels%22,%22refId%22:%22A%22%7D%7D%7D%7D&orgId=1) BTW the newest promtail appears to be 3.2.0.

IMHO -7.105427357601002e-15 is zero; the bug is in the tolerance.

@fpletz I concur there is a problem, ~however I'm not sure it is architecture-specific~. The "bigzero" test will work for some orderings of series and fail for others. Different orderings...

Hi, thanks for your interest. If you are coming up to speed, the first step would be to get familiar with Prometheus and recording rules. Then, to start with you...

OK, that's great. I think this is simple enough that you can just make a PR; no design doc required. But if you've thought about it and want to propose...

> Might be useful for other operations (WAL writing? Would this require holding WAL records in memory until there is 4KB to write? That gives me mixed feelings.

Adding the metric `container_memory_usage_bytes`, which a lot of people look at, to the Prombench dashboard we see that it is significantly lower with Direct IO.