Match with exact metric name does not work

[ar3s3ru]

Hello all :wave:

I've recently integrated statsd_exporter into a legacy service that is still using StatsD metrics, and I noticed that matching using the exact metric name doesn't seem to be exported under /metrics.

This is my mapping.yaml file:

mappings:
  # The "metric.*.name" + "match_type:regex" matching is the hack I found to be working.
  # In all these cases I would like to use exact matches, but I can't because the exported
  # won't export it under /metrics.
  - match: "Service.*.Http.request-count"
    match_type: regex
    name: "service_http_requests_count"
    help: "Total number of HTTP requests"

  - match: "Service.*.Http.response-duration-milliseconds"
    match_type: regex
    observer_type: histogram
    histogram_options:
      buckets: [.1, .25, .5, 1, 2.5, 5, 10]
    name: "service_http_responses_duration_seconds"
    help: "Histogram of HTTP response latencies"

  # This metric is not mapped to /metrics.
  - match: "subscription.delivery.pauses"
    name: "service_delivery_pauses_count"

  - match: "."
    match_type: regex
    action: drop
    name: "dropped"

These are the metrics values in /metrics related to the exporter. I was hoping to find some useful information about this behavior but I could not... It looks like the mapping is loaded (statsd_exporter_loaded_mappings 4) but the exact-match one (subscription.delivery.pauses) is not counted in statsd_exporter_metrics_total{type="counter"}, nor I can see anything in statsd_exporter_events_unmapped_total.

# HELP statsd_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which statsd_exporter was built.
# TYPE statsd_exporter_build_info gauge
statsd_exporter_build_info{branch="HEAD",goversion="go1.14.7",revision="bac6cbe8c5f745ea5a66033727dd1420a41d31dc",version="0.18.0"} 1
# HELP statsd_exporter_event_queue_flushed_total Number of times events were flushed to exporter
# TYPE statsd_exporter_event_queue_flushed_total counter
statsd_exporter_event_queue_flushed_total 3404
# HELP statsd_exporter_events_actions_total The total number of StatsD events by action.
# TYPE statsd_exporter_events_actions_total counter
statsd_exporter_events_actions_total{action="drop"} 1181
statsd_exporter_events_actions_total{action="map"} 72
# HELP statsd_exporter_events_total The total number of StatsD events seen.
# TYPE statsd_exporter_events_total counter
statsd_exporter_events_total{type="counter"} 36
statsd_exporter_events_total{type="observer"} 36
# HELP statsd_exporter_events_unmapped_total The total number of StatsD events no mapping was found for.
# TYPE statsd_exporter_events_unmapped_total counter
statsd_exporter_events_unmapped_total 0
# HELP statsd_exporter_lines_total The total number of StatsD lines received.
# TYPE statsd_exporter_lines_total counter
statsd_exporter_lines_total 1253
# HELP statsd_exporter_loaded_mappings The current number of configured metric mappings.
# TYPE statsd_exporter_loaded_mappings gauge
statsd_exporter_loaded_mappings 4
# HELP statsd_exporter_metrics_total The total number of metrics.
# TYPE statsd_exporter_metrics_total gauge
statsd_exporter_metrics_total{type="counter"} 1
statsd_exporter_metrics_total{type="histogram"} 1
# HELP statsd_exporter_samples_total The total number of StatsD samples received.
# TYPE statsd_exporter_samples_total counter
statsd_exporter_samples_total 1253
# HELP statsd_exporter_tag_errors_total The number of errors parsing DogStatsD tags.
# TYPE statsd_exporter_tag_errors_total counter
statsd_exporter_tag_errors_total 0
# HELP statsd_exporter_tags_total The total number of DogStatsD tags processed.
# TYPE statsd_exporter_tags_total counter
statsd_exporter_tags_total 73
# HELP statsd_exporter_tcp_connection_errors_total The number of errors encountered reading from TCP.
# TYPE statsd_exporter_tcp_connection_errors_total counter
statsd_exporter_tcp_connection_errors_total 0
# HELP statsd_exporter_tcp_connections_total The total number of TCP connections handled.
# TYPE statsd_exporter_tcp_connections_total counter
statsd_exporter_tcp_connections_total 0
# HELP statsd_exporter_tcp_too_long_lines_total The number of lines discarded due to being too long.
# TYPE statsd_exporter_tcp_too_long_lines_total counter
statsd_exporter_tcp_too_long_lines_total 0
# HELP statsd_exporter_udp_packets_total The total number of StatsD packets received over UDP.
# TYPE statsd_exporter_udp_packets_total counter
statsd_exporter_udp_packets_total 1253
# HELP statsd_exporter_unixgram_packets_total The total number of StatsD packets received over Unixgram.
# TYPE statsd_exporter_unixgram_packets_total counter
statsd_exporter_unixgram_packets_total 0
# HELP statsd_metric_mapper_cache_gets_total The count of total metric cache gets.
# TYPE statsd_metric_mapper_cache_gets_total counter
statsd_metric_mapper_cache_gets_total 1253
# HELP statsd_metric_mapper_cache_hits_total The count of total metric cache hits.
# TYPE statsd_metric_mapper_cache_hits_total counter
statsd_metric_mapper_cache_hits_total 1092
# HELP statsd_metric_mapper_cache_length The count of unique metrics currently cached.
# TYPE statsd_metric_mapper_cache_length gauge
statsd_metric_mapper_cache_length 161

Got any idea? Thanks :pray:

[matthiasr]

Interesting, yeah, this is a bug to me. My guess is that the FSM has an interesting edge case when everything is fixed?

[ar3s3ru]

That could it, but looking through the test cases in pkg/mapper/mapper_test.go I can see a fixture that resembles my particular case:

https://github.com/prometheus/statsd_exporter/blob/8b2b4c1a2b49a5a2a8c0c8c567462affd79b58d3/pkg/mapper/mapper_test.go#L166-L169

Also I noticed there aren't many tests for pkg/mapper/fsm, so it might very well be an FSM bug.

[matthiasr]

That test is not the same, it covers the case where a metric could be matched both by a fixed message and a glob. I am also confused by your other matches – they use match_type: regex but the first two look more like a glob match?

[glightfoot]

I'm curious if removing the last mapping that matches everything would result in it being captured