[debugPrintProcessor]: exit directly when log doesn't have debug level enabled

[pkoutsovasilis]

Proposed commit message

This PR is the aftermath of profiling the netflow integration test of this PR (PS: the pcap I used as input is ~25MB thus I won't include it in beats repo)

So this is the cpu profile flamegraph before the commit of this PR

As we can see debugPrintProcessor is invoked and we pay the "performance" price of encoding every event in json even when our log isn't set up with DEBUG level.

The respective cpu profile flamegraph after this PR with debugPrintProcessor still being there but exiting directly when log DEBUG is not enabled

Extracting the average EPS of the aformentioned netflow integration tested on my local macbook with a local ES inside a container

Before this PR	With this PR
18490	33760

Checklist

• [x] My code follows the style guidelines of this project
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] I have made corresponding change to the default configuration files
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

N/A

Author's Checklist

N/A

How to test this PR locally

Help get this PR merged or clone it and apply this small commit there 🙂

cd x-pack/filebeat
mage docker:composeUp

run go profiler against TestNetFlowIntegration

Related issues

• Relates https://github.com/elastic/beats/issues/37761

Use cases

N/A

Screenshots

Look at description

Logs

N/A

[pkoutsovasilis]

Good find!

IIUC this issue has been affecting agent since 8.7.0?

after looking around the git history, I think that the PR which caused this is this one as the older implementation had the encoding part under an if which was checking the log level. So only 8.15.0 is affected?!

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[pierrehilbert]

Thanks for catching that @pkoutsovasilis. @belimawr could you please have a look here as this is related to your change?

[cmacknz]

For Elastic Agent this is 8.15.0 only, for standalone Beats this has been there since the beginning when it was first implemented: https://github.com/elastic/beats/commit/83dfb2f7b7b411cb266682572a6691c221f903b8

When under agent it was gated by: https://github.com/elastic/beats/blob/cef7b39ee75b369eaaa9c1f992d151731c6da6d3/libbeat/management/agent.go#L50-L62

[cmacknz]

CC @alexsapran I would expect this to show up in the Filebeat benchmarks.

[alexsapran]

CC @alexsapran I would expect this to show up in the Filebeat benchmarks.

You would be right to expect that. Indeed, once I could run the elastic-agent benchmark again and review the generated diagnostics, the frame debugPrintProcessor was shown.

Unfortunately, this work is not yet complete. Running elastic-agent benchmarks was just unblocked from @pkoutsovasilis some days ago. The end goal is to integrate this into CI. So, looking at the future, it is possible to catch them.

[pkoutsovasilis]

with the opportunity of this PR I think it is a good idea to investigate if we can help in adding support to benchmark Filebeat inputs owned by @elastic/sec-deployment-and-devices

@pierrehilbert @cmacknz should we wait for a review from @belimawr ?

[pierrehilbert]

@belimawr will soon start his day so we can wait and get this merged today after his review if this is okay for you?