Filebeat Processors - make uppercase and lowercase processors (as in ES ingest nodes) available to filebeat

[ccmsi]

Greetings and thanks for this software.

I would like to request for adding the two processors uppercase and lowercase to filebeat input processing.

Besides being useful, adds to making things consistent.

Thanks for your time and attention

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[andresrc]

We are encouraging the use of ingest-node processors. What's the driving use case to have them available in the beats themselves?

[ccmsi]

The question seems like asking about personal taste. And as there are 2 options that's in a way how it is.

However, its filebeat that offers that option to begin with. There are processors. So one question in return is, why then does it exist if that is your question? IMO filebeat team by implementing processors has already expressed that interest for it to be there and as such this question seems awkward.

For support, i appreciate the decision of the filebeat team to provide processors. I think central management is nice, but distributing load is advantageous performance wise and offers flexibility. Changes are easier to work on while reducing risk.

My request though is about why not having those two simple ones, where the more complex ones, even script, are available.

[trnl]

I would double that.

Now we have hideous rename block with a lot of boilerplate because of casing:

          - from: header.x-request-id
            to: x-request-id
          - from: headers.x-request-id
            to: x-request-id
          - from: headers.X-Request-ID
            to: x-request-id
          - from: headers.X-Request-Id
            to: x-request-id
          - from: headers.X-request-Id
            to: x-request-id

[bduffy-aylien]

Hi, lower and upper case processors would be nice. You can achieve something similar just using the script processor. For example I have a field for log level and I can set it to lowercase like so;

        -
          script:
            lang: javascript
            id: lowercase
            source: >
              function process(event) {
                  var level = event.Get("level");
                  if(level != null) {
                      event.Put("level", level.toString().toLowerCase());
                  }
              }

[lplazas]

If no one is working on this one I'd like to give it a go as my first issue in this repo

[mr1716]

Hey, how is this progressing?? Is there any sense of a release date for adding the uppercase and lowercase processors in a filebeat release?

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[sqall01]

I would also like to know the state of this issue.

We need case-insensitive filtering on Winlogbeat (without using regex). And this processor would allow us to build it.

[mr1716]

@sqall01 there is the potential to use the script processor and the Javascript toLowerCase and toUpperCase to do this, but this requires looping through the desired values, deleting the old values, and then replacing with the new values. So having this done by a processor would be really helpful! Maybe this is something that elastic can build in for us?

[sqall01]

@mr1716 yes the script processor would be a possibility. However, we have to do this for a lot of events that we filtering and I have concerns regarding the performance if we always go through Javascript for this.

[mr1716]

@sqall01 agreed. Just throwing it out there as an option, even though it isnt equivalent

[mr1716]

@lplazas How is development going?? May you please provide an update?

[mr1716]

@jlind23 Any possibility to get some engineers to assist with this development?

[jlind23]

@mr1716 this is something that should go through our prioritisation first, for now we do not have someone to work on it.

[mr1716]

@jlind23 how would that take place and are there actions that the community would need to do?

[jlind23]

@mr1716 The only thing I can think of right now is that this is something you can work on if you need and i'll find someone to review and assist.

[mr1716]

@jlind23 Ok. I'll see if there is time, but there looks to be the need. I have a way to do it in Javascript, so maybe the employee can take what I have and convert it to something that can be converted into Filebeat?

[jlind23]

@mr1716 it will still imply too much time availability on their end. I will discuss with @nimarezainia on it first.

[mr1716]

@jlind23 hey, hope all is well. IS there any progress or update to provide?

[jlind23]

@mr1716 nothing occurred as we were focused on other areas. Let me add it to @nimarezainia and @pierrehilbert's plate for awareness and prioritization.

[hartfordfive]

Is there a plan yet to implement this feature in the near future?

[zez3]

I think there are some discussions but nothing concrete https://github.com/elastic/elastic-agent-shipper/issues/160

[zez3]

and this https://github.com/elastic/elastic-package/issues/949#issuecomment-1262753167 "we will try and pursiue the correct implementation in 8.7 (once prioritized amongst other asks there)"

[mr1716]

@zez3 thanks for this. So this means that there will be the implementation at some point, which is great!