always_include_global_tags depends on config file naming/order

[1tft]

Relevant telegraf.conf

# general config file telegraf.conf
[global_tags]
  globaltag = "globaltagvalue"

[agent]
  always_include_global_tags = true

Relevant myconfig.conf

# custom config file myconfig.conf
[[inputs.mock]]
  metric_name = "mock"
  taginclude = [ "mylocaltag" ]

 [[inputs.mock.constant]]
   name = "constant"
   value = "myvalue"

 [inputs.mock.tags]
   mylocaltag = "mylocaltagvalue"

Logs from Telegraf

[...]
2024-11-25T11:21:40Z I! Loading config: /etc/telegraf/telegraf.conf
2024-11-25T11:21:40Z I! Loading config: /etc/telegraf/telegraf.d/myconfig.conf
[...]

vs.

[...]
2024-11-25T11:23:13Z I! Loading config: /etc/telegraf/telegraf.d/myconfig.conf
2024-11-25T11:23:13Z I! Loading config: /etc/telegraf/telegraf.conf
[...]

System info

Telegraf 1.32.1

Docker

No response

Steps to reproduce

telegraf --config /etc/telegraf/telegraf.d/telegraf.conf --config /etc/telegraf/myconfig.conf --test produces a different/correct result > mock,globaltag=globaltagvalue,host=abc,mylocaltag=mylocaltagvalue constant="myvalue" 1732533700000000000 than telegraf --config /etc/telegraf/myconfig.conf --config /etc/telegraf/telegraf.d/telegraf.conf --test > mock,mylocaltag=mylocaltagvalue constant="myvalue" 1732533794000000000

Expected behavior

We expect same result, independent from config file name/order, because when you write same config in only one global file it is working correctly:

[[inputs.mock]]
  metric_name = "mock"
  taginclude = [ "mylocaltag" ]

 [[inputs.mock.constant]]
   name = "constant"
   value = "myvalue"

 [inputs.mock.tags]
   mylocaltag = "mylocaltagvalue"


[global_tags]
  globaltag = "globaltagvalue"

[agent]
  always_include_global_tags = true

Actual behavior

Telegraf produces differents results depending on order of config file reading/naming.

Additional info

Outside of --test modus you dont see this error, when you use general /etc/telegraf/telegraf.conf and custom configs inside /etc/telegraf/telegraf.d(*.conf, because telegraf.conf is always loaded at beginning. So its not so important for us, to fix this bug soon.

[knollet]

loadConfig(): https://github.com/influxdata/telegraf/blob/master/config/config.go#L511 is called for every config file. If the [agent] section with always_include_global_tags=true is in the second file loaded, every [[inputs...]]-plugin instantiated in the first config file gets the default (= false) always_include_global_tags. (loadConfig() -> addInput() -> buildInput() -> config.go Line 1521)

One could ask why the default tags and always_include_global_tags are "snapshotted" into the seperate input configs instead of remaining a global variable/constant.

---

Thought: Removing AlwaysIncludeTags from input object config instances and just using it as a global variable would change the input configs hash, would it not? So state saved in statefiles would be lost?

Thought2: @srebhan - we talked about statefiles being droppable in case of error: Isn't state always dropped if telegraf is updated to a new version and the plugin is changed in a way? For example, if there's a new (but unused, remaining on default setting) config option. Also: Is state dropped if there's a new global tag? (so state is dropped by nonlocal (i.e. another file) changes in the config?)

[srebhan]

Sorry for the late reply on this topic! The problem currently occurs for all global variables like in the agent or global_tags sections. I'm working on a restructuring of the configuration process but given the degree of intrusiveness this is not planned to land anytime soon. It is therefor recommended to put the agent and global_tags sections into the /etc/telegraf.conf file (or separate files) and explicitly specify those via --config before the files (or config directory) containing the plugins as the loading order respects the order of the command line parameters.

[srebhan]

@knollet regarding

Thought2: @srebhan - we talked about statefiles being droppable in case of error: Isn't state always dropped if telegraf is updated to a new version and the plugin is changed in a way? For example, if there's a new (but unused, remaining on default setting) config option. Also: Is state dropped if there's a new global tag? (so state is dropped by nonlocal (i.e. another file) changes in the config?)

We should probably open a new issue or have a chat on Slack. Anyway, if a new but unused (i.e. commented) option is added to the configuration the plugin ID should not change. Only if the user specifies the new option. If the ID changes that might indicate a bug. For the global tags (or other global options), those are not taken into account for the ID computation of therefor will reset the state.

[1tft]

Thank you for explaining this issue. We will try to keep this known issue in mind. Therefore, it is ok for us to close this ticket.

[srebhan]

Let's keep this open for others as a reference. We really need to fix this on the long run...