metrics_agent plugin does not produce any metrics for non-output plugins

[mlasevich]

Describe the bug I recently noticed that there is no usable data in output of the monitor_agent plugin for any non-output plugins (except for retry_count)

To Reproduce docker-compose.yaml:

version: "3"

services:
  fluentd:
    image: fluent/fluentd:v1.11-2
    volumes:
      - ./fluent.conf:/fluentd/etc/fluent.conf

fluent.conf:

<source>
  @type dummy
  @id in_dummy
  dummy {"hello":"world"}
  auto_increment_key count
  rate 1
  tag dummy
</source>

<source>
  @type monitor_agent
  @id in_monitor
  tag stats.fluentd
  emit_interval 10
</source>

<match **>
  @type stdout
  @id out_stdout
</match>

output:

2020-10-13 20:41:07.002072200 +0000 dummy: {"hello":"world","count":19}
2020-10-13 20:41:07.500727200 +0000 stats.fluentd: {"plugin_id":"in_dummy","plugin_category":"input","type":"dummy","output_plugin":false,"retry_count":null}
2020-10-13 20:41:07.500727200 +0000 stats.fluentd: {"plugin_id":"in_monitor","plugin_category":"input","type":"monitor_agent","output_plugin":false,"retry_count":null}
2020-10-13 20:41:07.500727200 +0000 stats.fluentd: {"plugin_id":"out_stdout","plugin_category":"output","type":"stdout","output_plugin":true,"retry_count":0,"emit_records":25,"emit_count":23,"write_count":0,"rollback_count":0,"slow_flush_count":0,"flush_time_count":0}

Expected behavior

Expect to see number of records received/processed and other useful statistics on all emitted records, not just output one

Your Environment

For sake of testing, used docker image fluent/fluentd:v1.11-2 - but same seems to be happening in multiple versions

If you hit the problem with older fluentd version, try latest version first.

Your Configuration

(see above )

Your Error Log

(see above )

[kenhys]

@mlasevich It seems that it is the intended behavior of fluentd. The response may change but it depends on plugin. ref. https://github.com/fluent/fluentd/blob/v1.11.2/lib/fluent/plugin/in_monitor_agent.rb#L238

FYI: you may able to get further information by http://127.0.0.1:24220/api/plugins.json?debug=1 query.

[mlasevich]

Yeah, seems that entire monitoring plugin is completely useless outside the output plugins. (and even that seems to be an afterthought, added later) - not really sure why it was created in the first place. Must have been a placeholder for something never completed :-(

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[repeatedly]

Remove stale and add feature request. We need more feedback which metrics are needed for other plugin types.

[stevyn81]

Hello, for input plugins, is the retry_count always null or can it change during trouble? Is it worth monitoring?