Fluentd custom metrics are not working

[trK54Ylmz]

I've put print outs to the instrument method at 162nd line, https://github.com/fluent/fluent-plugin-prometheus/blob/46483296ecc00c03985c56cfa23d38e4454e78fd/lib/fluent/plugin/prometheus.rb#L162

puts labels(record, expander, placeholders)
puts value

@counter.increment(labels(record, expander, placeholders), value)

here is the output,

$ fluentd -c /etc/fluent/fluent.conf

2019-01-16 11:51:22.000000000 +0300 app.test: {"test":1,"number":2958}
{:tag=>"app.test", :worker=>"0", :hostname=>"fluentd-test"}
1
2019-01-16 11:51:24.000000000 +0300 app.test: {"test":1,"number":2959}
{:tag=>"app.test", :worker=>"0", :hostname=>"fluentd-test"}
1
2019-01-16 11:51:26.000000000 +0300 app.test: {"test":1,"number":2960}
{:tag=>"app.test", :worker=>"0", :hostname=>"fluentd-test"}
1
2019-01-16 11:51:28.000000000 +0300 app.test: {"test":1,"number":2961}
{:tag=>"app.test", :worker=>"0", :hostname=>"fluentd-test"}
1
2019-01-16 11:51:30.000000000 +0300 app.test: {"test":1,"number":2962}
{:tag=>"app.test", :worker=>"0", :hostname=>"fluentd-test"}
1
2019-01-16 11:51:32.000000000 +0300 app.test: {"test":1,"number":2963}

seems ok right? but metrics endpoint is different,

# TYPE fluentd_test_input_status_num_records_total counter
# HELP fluentd_test_input_status_num_records_total The total number of incoming records

configuration

<system>
  workers 8                    # also can be specified by `--workers` command line option
  root_dir /etc/fluent         # strongly recommended to specify this
</system>

<source>
  @type forward
  port 24200
</source>

<source>
  @type monitor_agent
  @id monitor_agent_input

  port 24240
</source>

<source>
  @type prometheus
  bind 0.0.0.0
  port 24231
  metrics_path /metrics
</source>

<filter app.**>
  @type prometheus
  <metric>
    name fluentd_test_input_status_num_records_total
    type counter
    desc The total number of incoming records
    <labels>
      tag ${tag}
      worker ${worker_id}
      hostname ${hostname}
    </labels>
  </metric>
</filter>

<match app.**>
    @type stdout
</match>

gems

$ gem list
fluent-plugin-prometheus (1.3.0)
fluentd (1.3.3)
prometheus-client (0.8.0)