Old metrics replaced with '-1' when used with report_filename option

[dev-e]

When the option report_filename in prometheus.yaml is enabled, any new puppet agent launch triggers rewrite of "old" metrics with -1 values. It is absolutely not valid because we lose data about other hosts that launched puppet before the last host. What is the value of this replacement?

# HELP puppet_report_resources Resources broken down by their state
# TYPE puppet_report_resources gauge
# HELP puppet_report_time Resource apply times
# TYPE puppet_report_time gauge
# HELP puppet_report_changes Changed resources in the last puppet run
# TYPE puppet_report_changes gauge
# HELP puppet_report_events Resource application events
# TYPE puppet_report_events gauge
# HELP puppet_report Unix timestamp of the last puppet run
# TYPE puppet_report gauge
# Old metrics
puppet_report_resources{name="Changed",environment="prod",host="previous_host"} -1
puppet_report_resources{name="Corrective change",environment="prod",host="previous_host"} -1
puppet_report_resources{name="Failed",environment="prod",host="previous_host"} -1
puppet_report_resources{name="Failed to restart",environment="prod",host="previous_host"} -1
puppet_report_resources{name="Out of sync",environment="prod",host="previous_host"} -1
puppet_report_resources{name="Restarted",environment="prod",host="previous_host"} -1
puppet_report_resources{name="Scheduled",environment="prod",host="previous_host"} -1
puppet_report_resources{name="Skipped",environment="prod",host="previous_host"} -1
puppet_report_resources{name="Total",environment="prod",host="previous_host"} -1
puppet_report_time{name="Anchor",environment="prod",host="previous_host"} -1
puppet_report_time{name="Apt key",environment="prod",host="previous_host"} -1
puppet_report_time{name="Archive",environment="prod",host="previous_host"} -1
puppet_report_time{name="Catalog application",environment="prod",host="previous_host"} -1
puppet_report_time{name="Concat file",environment="prod",host="previous_host"} -1
puppet_report_time{name="Concat fragment",environment="prod",host="previous_host"} -1
puppet_report_time{name="Config retrieval",environment="prod",host="previous_host"} -1
puppet_report_time{name="Convert catalog",environment="prod",host="previous_host"} -1
puppet_report_time{name="Cron",environment="prod",host="previous_host"} -1
puppet_report_time{name="Exec",environment="prod",host="previous_host"} -1
puppet_report_time{name="Fact generation",environment="prod",host="previous_host"} -1
puppet_report_time{name="File",environment="prod",host="previous_host"} -1
puppet_report_time{name="Filebucket",environment="prod",host="previous_host"} -1
puppet_report_time{name="Group",environment="prod",host="previous_host"} -1
puppet_report_time{name="Httpauth",environment="prod",host="previous_host"} -1
puppet_report_time{name="Node retrieval",environment="prod",host="previous_host"} -1
puppet_report_time{name="Package",environment="prod",host="previous_host"} -1
puppet_report_time{name="Plugin sync",environment="prod",host="previous_host"} -1
puppet_report_time{name="Resources",environment="prod",host="previous_host"} -1
puppet_report_time{name="Schedule",environment="prod",host="previous_host"} -1
puppet_report_time{name="Service",environment="prod",host="previous_host"} -1
puppet_report_time{name="Total",environment="prod",host="previous_host"} -1
puppet_report_time{name="Transaction evaluation",environment="prod",host="previous_host"} -1
puppet_report_time{name="User",environment="prod",host="previous_host"} -1
puppet_report_changes{name="Total",environment="prod",host="previous_host"} -1
puppet_report_events{name="Failure",environment="prod",host="previous_host"} -1
puppet_report_events{name="Success",environment="prod",host="previous_host"} -1
puppet_report_events{name="Total",environment="prod",host="previous_host"} -1
puppet_report{environment="prod",host="previous_host"} -1
# New metrics
puppet_report_resources{name="Changed",environment="prod",host="latest_host"} 1
puppet_report_resources{name="Corrective change",environment="prod",host="latest_host"} 1
puppet_report_resources{name="Failed",environment="prod",host="latest_host"} 0
puppet_report_resources{name="Failed to restart",environment="prod",host="latest_host"} 0
puppet_report_resources{name="Out of sync",environment="prod",host="latest_host"} 1
puppet_report_resources{name="Restarted",environment="prod",host="latest_host"} 0
puppet_report_resources{name="Scheduled",environment="prod",host="latest_host"} 0
puppet_report_resources{name="Skipped",environment="prod",host="latest_host"} 0
puppet_report_resources{name="Total",environment="prod",host="latest_host"} 1419
puppet_report_time{name="Anchor",environment="prod",host="latest_host"} 0.00020055399999999998
puppet_report_time{name="Apt key",environment="prod",host="latest_host"} 0.0010214339999999999
puppet_report_time{name="Archive",environment="prod",host="latest_host"} 0.0013130680000000001
puppet_report_time{name="Catalog application",environment="prod",host="latest_host"} 12.731362691149116
puppet_report_time{name="Concat file",environment="prod",host="latest_host"} 0.000569588
puppet_report_time{name="Concat fragment",environment="prod",host="latest_host"} 0.0018003480000000002
puppet_report_time{name="Config retrieval",environment="prod",host="latest_host"} 7.667529137805104
puppet_report_time{name="Convert catalog",environment="prod",host="latest_host"} 1.1087825242429972
puppet_report_time{name="Cron",environment="prod",host="latest_host"} 0.021340405
puppet_report_time{name="Exec",environment="prod",host="latest_host"} 2.460908792999999
puppet_report_time{name="Fact generation",environment="prod",host="latest_host"} 6.378860469907522
puppet_report_time{name="File",environment="prod",host="latest_host"} 4.624337396000001
puppet_report_time{name="Filebucket",environment="prod",host="latest_host"} 6.5871e-05
puppet_report_time{name="Group",environment="prod",host="latest_host"} 0.13283111100000008
puppet_report_time{name="Httpauth",environment="prod",host="latest_host"} 0.000573358
puppet_report_time{name="Node retrieval",environment="prod",host="latest_host"} 0.3272962998598814
puppet_report_time{name="Package",environment="prod",host="latest_host"} 0.4034370560000001
puppet_report_time{name="Plugin sync",environment="prod",host="latest_host"} 1.0822257678955793
puppet_report_time{name="Resources",environment="prod",host="latest_host"} 8.5261e-05
puppet_report_time{name="Schedule",environment="prod",host="latest_host"} 0.00038971599999999997
puppet_report_time{name="Service",environment="prod",host="latest_host"} 0.54392664
puppet_report_time{name="Total",environment="prod",host="latest_host"} 29.313116721
puppet_report_time{name="Transaction evaluation",environment="prod",host="latest_host"} 12.170660078525543
puppet_report_time{name="User",environment="prod",host="latest_host"} 0.19735986999999997
puppet_report_changes{name="Total",environment="prod",host="latest_host"} 1
puppet_report_events{name="Failure",environment="prod",host="latest_host"} 0
puppet_report_events{name="Success",environment="prod",host="latest_host"} 1
puppet_report_events{name="Total",environment="prod",host="latest_host"} 1
puppet_report{environment="prod",host="latest_host"} 1585743103.709

[dev-e]

Any updates? Or you recommend not to use report_filename and use separate metrics file for every puppet host?

[theobarrague]

UP, got error with that

Oct 14 16:10:51 inf-puppetmaster01-p03 prometheus-node-exporter[128270]: ts=2022-10-14T14:10:50.998Z caller=stdlib.go:105 level=error msg="error gathering metrics: 167 error(s) occurred:\n* [from Gatherer #2] collected metric puppet_report_resources label:<name:\"environment\" value:\"staging\" > label:<name:\"host\" value:\"brs-apidspcachehttp01-p01.dc01\" > label:<name:\"name\" value:\"Changed\" > untyped:<value:-1 >  has help \"Metric read from /var/lib/prometheus/node-exporter/...\" but should have ...