Add option to add additional info to internal statistics

[Hipska]

Use Case

Internal statistics related to an input plugin only add alias or _id as additional info to identify the plugin when they are set in the config. This is mostly not enough to query them afterwards in the database.

An option to also add the labels or tags configured on the plugin to these statistics would help.

Example

I have an inputs.snmp for each device I collect info from. They are grouped by addition of a role tag. I also added the device name as alias value to have distinct logs and internal_gather metrics.

[[inputs.snmp]]
  alias = "fw001"
  agents = ["10.0.0.1:161"]

  [inputs.snmp.tags]
    role = "firewall"

When I want to see error counters from all the firewalls, then I need to do the following:

1. Gather all hostnames where role equals firewall.
2. Prepare a huge query with all the retrieved names: (pseudo code)

SELECT errors FROM internal_gather
WHERE input = 'snmp' AND REGEXP_LIKE(alias, '^(fw001|fw002|fw003|...)$');

You can imagine this could be a lot more performant when a query like this would be possible instead:

SELECT errors FROM internal_gather
WHERE input = 'snmp' AND role = 'firewall';

Expected behavior

A global config option to enable addition of plugin-level defined tags or labels:

- internal_gather,alias=fw001,input=snmp,version=1.36.4 errors=0i,gather_time_ns=0i,gather_timeouts=0i,metrics_gathered=0i
+ internal_gather,alias=fw001,input=snmp,role=firewall,version=1.36.4 errors=0i,gather_time_ns=0i,gather_timeouts=0i,metrics_gathered=0i

Actual behavior

No additional info on internal plugin metrics.

Additional info

Related spec TSD-011 Internal plugin statistics collection says:

The injected collector instance must add all relevant model-level information such as an optional alias setting or tags settings.

However, it was requested in https://github.com/influxdata/telegraf/pull/17749#discussion_r2414293656 to remove tags copy, rendering the current implementation to not adhere to the spec.

[Hipska]

@srebhan Any feedback? Has this been discussed with the team? IMO telegraf still doesn't correctly implement TSD-011.

Should I just create the PR and see feedback on there?

[srebhan]

Yeah the spec says that but this would break backward compatibility if you add as-is. What we can do is to add an option to the internal input similar to the per_instance option, e.g. calling it with_individual_tags and accumulate the information over different tags if not set... What do you think?