Provide a "SED-like" mechanism to "find and replace" characters in a log record.

[matthewmodestino]

trafficstars

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

As an OTel native logging user, I need a way to "find and replace" characters that may exist in the original log record that I don't want/need in the logging backend. This can be useful to anonymize info, but also to make log records more usable in logging backends.

My current example is removing ANSI colour codes from my log files...

[2022-01-04 01:51:50] [36mINFO[39m "GET /ads.txt" [36m301[39m 10ms

Describe the solution you'd like A clear and concise description of what you want to happen.

I want to define a SED command processor to find and replace the ANSI colour codes, or any other characters in the log record, as I see fit. Something like:

SEDCMD-removecolorcodes = s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

I looked through existing OTel processors but I only see REGEX based ones, and none that make it easy to achieve "find and replace" in existing log pipelines

Additional context Add any other context or screenshots about the feature request here.

Sometimes kubernetes or application logs will use colour codes to make pretty printing when viewed in a terminal, or may include strings that need to be anonymized or replaced to make the log more usable by the user in the logging backend.

[jpkrohling]

@bogdandrutu, @anuraaga, is this something the transform processor should take care of?

[matthewmodestino]

Bump.

Any progress on transform processor?

Would be really handy to have the if style expressions that the log collection operators support.

[anuraaga]

Hi @jpkrohling sorry for not responding to the question - yes replacement, possibly with SED syntax or something equivalent, is a function we'd want in the transform processor.

@matthewmodestino Sorry for the delay, just sent #8972 with a big step in being able to add logs support

[matthewmodestino]

Sweet!

In the logs signal, we would likely need to apply this conditionally,ie. based on resource or attribute values. Looking forward to seeing how this turns out!

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[matthewmodestino]

Bump. Still need this.

[jpkrohling]

@kovrus, @TylerHelmuth, is this doable today with OTTL?

[github-actions[bot]]

Pinging code owners: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

@matthewmodestino checkout the transform processor and the function replace_pattern. Does that fit your needs?

[kovrus]

👍 replace_pattern(body, regex, "") probably can address that.

[matthewmodestino]

Will give it a shot! Thanks!

[TylerHelmuth]

@matthewmodestino I'm going to close this based on the transform processor's capabilities. Please ping me if you think it should stay open.

[matthewmodestino]

Sounds good, been meaning to circle back and test. Will hit you up if need be. Thanks!