Allow Heroku logs containing dyno metrics to be transformed into useable attributes

[jdeff]

Component(s)

OTTL

Is your feature request related to a problem? Please describe.

Heroku emits log lines containing metrics data for the various dyno instances running for an application. These log lines look similar to:

source=web.1 dyno=heroku.1234-5678-9012-3456 sample#memory_total=10917.88MB sample#memory_rss=10849.69MB sample#memory_cache=68.20MB sample#memory_swap=0.00MB sample#memory_pgpgin=3384pages sample#memory_pgpgout=82pages sample#memory_quota=14336.00MB

Using the syslog_receiver and key_value_parser you can get something almost useable. However, the values are still strings of human readable byte sizes (e.g. "68.20MB") and need to converted to an integer number of bytes in order for them to be useable.

Describe the solution you'd like

I would like to be able to parse these byte size values using the transformprocessor, which I am already using to shape other values, and convert them to an integer value of bytes. I propose adding a ParseBytes function to OTTL in the same vain as ParseJson and ParseCsv. This could be used with the following config:

processors:
    transform:
      log_statements:
        - context: log
          statements:
            - replace_all_patterns(attributes, "key", "sample#memory-total", "system.memory.usage.used")
            - set(attributes["system.memory.usage.used"], ParseBytes(attributes["system.memory.usage.used"])) where attributes["system.memory.usage.used"] != nil

Describe alternatives you've considered

I've looked at simply striping the units from these values, but the units are not guaranteed to be the same.

ParseBytes may be an ambiguous or confusing name (though it is what is used by go-humanize). Something like ParsePrettyByteSize or ParseHumanizedByteString might be clearer.

Additional context

No response

[jdeff]

cc @TylerHelmuth

[github-actions[bot]]

Pinging code owners for processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

Pinging code owners for pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

Do you want those logs to be transformed into metric data?

[TylerHelmuth]

The ability to convert regular units to another unit is generally useful. I think a more generic function that takes an input and converts to a desired output would be most generally useful.

Warning, there are some weird edge cases with converting when you need to support stuff like mb, M, and Mi. It was so annoying we ended up writing a solution ourselves instead of using a library

[TylerHelmuth]

@jdeff can you propose what a function would look like to be able to convert 10917.88MB to any format?

[jdeff]

Do you want those logs to be transformed into metric data?

That might be something that is useful, but I don't necessarily need that in this case. I am able to chart the log data the same way in Honeycomb. Maybe there is another advantage that I am missing?

[TylerHelmuth]

@jdeff the ability to convert regular units to another unit is generally useful. Can you propose what a function would look like to be able to convert 10917.88MB to any format.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley @edmocosta
• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley @edmocosta

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

This issue has been closed as inactive because it has been stale for 120 days with no activity.