Enable string interpolation for hash-type parameters

[ChiliJohnson]

Is your feature request related to a problem? Please describe.

The FAQ mentions that you can use string interpolation to configure parameters using environment variables, e.g. some_field "#{ENV['FOO_HOME']}", but this doesn't seem to apply to keys or values in hash-type parameters.

Describe the solution you'd like

It would be nice if string values (or keys for that matter) in hash parameters could also get interpolated in the way string parameters already do. For example let's say I want to use an HTTP output with an authorization header that includes a token passed in through the environment:

<match>
  @type http
  headers {"Authorization": "Bearer #{ENV['SOME_ENV_VAR']}"}
</match>

Describe alternatives you've considered

For my particular example, a first-class feature to support bearer tokens in the HTTP output plugin (#3587) would likely solve my immediate problem, but also implementing more general-purpose string interpolation for hash parameters would remove the need for updates to that output plugin

Additional context

No response

[Athishpranav2003]

I was checking about this issue "{\"Authorization\":\"Bearer \#{ENV['TEST_PORT']}\"}" This is the format in which json.parse is invoked in https://github.com/fluent/fluentd/blob/master/lib/fluent/config/types.rb#L204

Ideally we would need "{\"Authorization\":\"Bearer #{ENV['TEST_PORT']}\"}" to be passed to json parser. I am checking if there is a clean way to make sure the # is not getting escaped for jsons

[Athishpranav2003]

For this there can be different ways in which this can be done

1. Instead of JSON.parse we use eval but this is dangerous and not advised as arbitrary code execution can happen
2. Manually extract the interpolated parts and substitute it with the values(seems reasonable)

For now i will work on the 2nd approach incase if there is different suggestion for same please ping me

@ashie @kenhys

[daipom]

@Athishpranav2003 Thanks! I'm seeing this.

Ideally we would need "{\"Authorization\":\"Bearer #{ENV['TEST_PORT']}\"}" to be passed to json parser. I am checking if there is a clean way to make sure the # is not getting escaped for jsons

I see. Looks like we need double quotes for the entire value.

<source>
  @type sample
  tag test
  sample "{\"k1\":\"v1\",\"k2\":\"#{ENV['FOO']}\"}"
</source>

<match test>
  @type stdout
</match>

$ FOO=foo bundle exec fluentd -c test.conf
...
2024-08-06 16:32:53.054415966 +0900 test: {"k1":"v1","k2":"foo"}
...

I think the issue is whether this could be made easier, am I right?

[Athishpranav2003]

Yah @daipom Either this PR will solve this or we change the FAQ in fluentd-docs to highlight about this case of string interpolation in case of hash and arrays alone

[daipom]

Thanks! I see! I'm seeing #4580.

[daipom]

We need to consider this as an issue of Embedded Ruby Code. Using environment variables is just one way of it.

Embedded Ruby Code feature is implemented in LiteralParser:

https://github.com/fluent/fluentd/blob/e763c0761c44d9734b6aa374371387a2e8406522/lib/fluent/config/literal_parser.rb#L183-L199

https://github.com/fluent/fluentd/blob/e763c0761c44d9734b6aa374371387a2e8406522/lib/fluent/config/literal_parser.rb#L108-L110

Since it is for double_quoted_string, we need to use double quotes for the entire value to use this feature.

It may be possible to address this as a types problem, as in #4580, but the specifications in LiteralParser need to be carefully considered.

[Athishpranav2003]

Also @daipom I haven't considered the case for hostname variable in string interpolation.

I tried looking literal_parser.rb, seems like scan_double_quoted_string is handling the env variable case but very specific to string input. Maybe a change in this code might be needed to handle the interpolated strings present in any other value of config file other than only in string types

[daipom]

The logic of literal_parser.rb is for the very early stage of parsing. At this stage, there is no need to consider the type of each config param.

The reason is unclear, but it seems that it only cares whether the string is JSON or not (regardless of the parameter type). I think it is because we want to parse the following multiline setting. However, It should be noted that JSON or string in LiteralParser is not related to the type of each config param.

<source>
  @type sample
  tag test
  sample {
          "k1":"v1",
          "k2":"v2"
         }
</source> 

<match test>
  @type stdout
</match>