Feature request: Pipeline IDs as variables

[ceeeekay]

I'd like to be able to use a specific pipeline's pipeline.id as a variable in a Logstash config - in this case to use it in the File output.

I have an external monitoring script that gets the pipeline IDs from the Logstash API, and checks a file with the same name to see if the heartbeat is current (from the heartbeat input). At the moment I have to assign these filenames manually, even though they're exactly the same as pipeline.id from pipelines.yml

Being able to use a variable for the filename would mean I can have the same output config for each pipeline, instead of manually naming each heartbeat file in a separate config for each pipeline.

Thanks!

[voiprodrigo]

This please!

[kwri-avongluck]

Yet another great idea left idle :-)

[andsel]

Looking at the code, the replacement of substitution variables with environment ones or the one from secret store happens at:

https://github.com/elastic/logstash/blob/2cf6675f538247be091df9d31cc455538f029a07/logstash-core/lib/logstash/util/substitution_variables.rb#L69-L73

Those replacement sources (ENV and SecretStores) are global for the entire Logstash process, to implement this feature a second level of replacement source should be introduced. This second level has a scope per pipeline.

[tibyke]

Any news on this?

[andsel]

This functionality could be implemented updating the org.logstash.StringInterpolation.evaluate static method to take count of a new special placeholder. logstash-output-file path parameter is already interpolated using Event.sprintfhttps://github.com/logstash-plugins/logstash-output-file/blob/70f77b9fb89c7c4628647c4b3902a4bc4088d62e/lib/logstash/outputs/file.rb#L176.

The interpolation logic at https://github.com/elastic/logstash/blob/b15c6c50fca607058338702c3db4a3ed05531c89/logstash-core/src/main/java/org/logstash/StringInterpolation.java#L115-L116 can be updated to handle this new placeholder.

The placeholder should be {{PIPELINE_ID}} so that it's usage could be something like

output {
  path => "/tmp/%{{PIPELINE_ID}}"
}