Feature Request: Support variables in fallback log group/stream names as well

[pranavmarla]

Rationale I know that default_log_group_name and default_log_stream_name were introduced specifically as fallbacks in case the variables in log_group_name and log_stream_name failed to parse, so I can understand the logic behind having the fallbacks just be "dumb strings", without any variable/templating support.

However, I think there is a benefit to having even the fallbacks support variables -- for example:

1. My first choice of log stream name is a dynamic variable that is NOT guaranteed to exist: optional-var
2. My fallback/second choice log stream name is ALSO a dynamic variable, but one that IS guaranteed to exist: Fluent Bit tag

Both these dynamic log stream names are preferable to a fixed, static name, which is currently my only fallback option.

[OUTPUT]
    Name                    cloudwatch
    Match                   *
    region                  us-east-1
    log_group_name          test-log-group
    auto_create_group       true
    log_stream_name         $(optional-var)
    default_log_stream_name $(tag)

Currently, in this case, when my primary variable (optional-var) doesn't exist, Fluent Bit literally creates a log stream named $(tag), instead of substituting in the Fluent Bit tag.

Edge Case Quick note on the obvious edge case here -- i.e. if the variable mentioned in the fallback/default log stream name also fails. At this point, given that the user has now had two chances to choose a variable that will always exist, I think it's reasonable to just throw a hard error and fail, rather than attempting to automatically recover somehow (or, god forbid, creating a new default_default_log_stream_name option), but that's just my suggestion.

[pranavmarla]

@davidnewhall Hope you can take a look at this when you get a chance

[Drupi]

Hello! Is it possible to also enable this? https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/record-accessor

[davidnewhall]

Hi @pranavmarla - I'm not directly involved with the firelens team at AWS, but I have provided some contributions to the fluent-bit go libraries. I'd happy to implement a feature request like this, but I'd need someone like @PettitWesley to give us some feedback on the implementation design.

I think we could probably allow N log group names in a slice and just parse through them until one works.

[pranavmarla]

Great, thank you @davidnewhall !

[PettitWesley]

@davidnewhall @pranavmarla I'm fine with the feature request... it'd be up the user to make sure they intelligently pick the fallback name... but I'm fine with that.

I think we could probably allow N log group names in a slice and just parse through them until one works.

That also works too possibly, I don't think Fluent Bit go plugins support anything but string parameters though so you'd have to have a delimiter like comma for the user input.

I'm also not certain we need that, simply having the fallback names also support variables (but warn the user to be smart) would be enough.

[PettitWesley]

@Drupi

Hello! Is it possible to also enable this? https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/record-accessor

Unfortunately that feature is not available to go plugins. However, the log stream/group name templating feature that David built supports the exact same things, just with a slightly different syntax.