logstash-filter-mutate icon indicating copy to clipboard operation
logstash-filter-mutate copied to clipboard

Lowercase mutate filter followed by gsub not working.

Open adison88 opened this issue 4 years ago • 2 comments

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

  • Version:
Logstash 6.8.5
Elasticsearch & Kibana 6.3
Filebeat 6.3
Winlogbeat 6.0
  • Operating System:
  • Config File (if you have sensitive info, please remove it):

` mutate { add_field => { "Static_Url" => "%{Url}" } }

    mutate {
        gsub => [
            "Static_Url", "\\/[({]?[a-fA-F0-9]{8}[-]?([a-fA-F0-9]{4}[-]?){3}[a-fA-F0-9]{12}[})]?+", "/{uuid}"
        ]
    }

    mutate {
        gsub => [
            "Static_Url", "\\/[0-9]+( \\/|\\/|$)", "/{id}/"
        ]
    }

    mutate {
        lowercase => [ "Static_Url" ]
    }

`

  • Sample Data:

` { "_type": "doc", "_version": 1, "_score": null, "_source": { "log_timestamp": "2021-03-29T02:31:28.000Z", "type": "lumberjack-json_line", "Url": "/DeviceServices/Dm.svc/token/jW6Q8/333CB41462686848A9BA73ECA4794B05", "path": "e:\AirWatch\Logs\IIS\W3SVC1\u_ex210329.log", "LogSize": 1432, "sc_bytes": "1097", "@version": "1", "beat": { "name": "", "version": "6.8.5", "hostname": "" }, "Static_Url": "/DeviceServices/Dm.svc/token/jW6Q8/{uuid}", "tags": [ "filebeat", "ws1", "iis-log", "us04_ls", "beats_input_codec_plain_applied", "lumberjack-json_line-5006" ], "delay": { "downstream_logstash_ack": -12.848, "filebeat_read": "2021-03-29T02:31:31.587Z" }, "@timestamp": "2021-03-29T02:31:31.587Z", "WinStatus": "0", "_logzio_pattern": 3516387, "Username": "-", "SubStatusCode": "0", "fields": { "filebeat_config_version": "20.12", "env_name": "CN1258", "ws1_release": "20.11.0.7" } }, "fields": { "log_timestamp": [ "2021-03-29T02:31:28.000Z" ], "@timestamp": [ "2021-03-29T02:31:31.587Z" ] }, "highlight": { "tags": [ "@kibana-highlighted-field@iis@/kibana-highlighted-field@-@kibana-highlighted-field@log@/kibana-highlighted-field@" ] }, "sort": [ 1616985091587 ] }

`

  • Steps to Reproduce:

I've 2 gsub filters followed by lowercase filter on newly added field. gsub filters work properly however lowercase filter does not work.

adison88 avatar Mar 29 '21 02:03 adison88

Thanks for the report, LS 6.3 is not longer supported. Could you try with latest LS 7.x if the issue still persist?

kares avatar Mar 29 '21 06:03 kares

@kares corrected the LS version is 6.8.5.

adison88 avatar Mar 30 '21 17:03 adison88