input_chunk: fix input plugin metrics not counting records re-emitted by filters downstream

[mindw]

move input metrics collections before applying filters.

Fixes #8729

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• [N/A] Debug log output from testing the change

• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [N/A] Documentation required for this feature

Backporting

• [x] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[patrick-stephens]

Should this be targeting the 3.0 or 3.1 branches? master is currently going to be the next 3.2 release: https://github.com/fluent/fluent-bit/wiki/Fluent-Bit-Roadmap

[mindw]

@patrick-stephens not sure how backporting works in the project. Typically master is first followed by backport PRs.

[patrick-stephens]

@patrick-stephens not sure how backporting works in the project. Typically master is first followed by backport PRs.

Yup, just trying to understand if this affects just 3.0.1 or more widely. I think you need to tick the backport checkbox then as well and ideally provide the related PRs for those backports.

[mindw]

@patrick-stephens not sure how backporting works in the project. Typically master is first followed by backport PRs.

Yup, just trying to understand if this affects just 3.0.1 or more widely. I think you need to tick the backport checkbox then as well and ideally provide the related PRs for those backports.

This affects 2.2.0+. 2.1.10 was the last version without this bug.

[edsiper]

checking on this

[edsiper]

@braydonk would you please check this PR since it seems related to the change introduced in https://github.com/fluent/fluent-bit/commit/1a8dd39e3a26ddf06d479d2314b96d51faa0dec9 ?

[braydonk]

This bug was introduced by me in https://github.com/fluent/fluent-bit/pull/8223 and https://github.com/fluent/fluent-bit/pull/8229. Because of the way the multiline and rewrite_tag Filter plugins work, the input_records and input_bytes metrics would end up getting recorded only for the emitter plugins associated with each of those plugins and not the original input source.

The reason I did that at the time was that this commit https://github.com/fluent/fluent-bit/commit/ac4ec1fe8db8c6f240b40afccaeaab8751ea0e1d at a glance looked like metrics were supposed to be recorded after writing to the input chunk. However that was a mistaken interpretation on my part at the time, and the important detail was that metrics were recorded before filters were applied.

Thank you @mindw for catching this, sorry for the inconvenience.

[mindw]

@braydonk @edsiper, thank you for the quick turn around. I believe the PRs' title should pass the checker.

[mindw]

@braydonk , @edsiper as CI is green across the board. Is anything left for me to do here?

[braydonk]

Approving the PR is the extent of my authority, it's up to maintainers from here.

[mindw]

@edsiper @braydonk it would be great to have this fix part the upcoming 3.2.0 release :)

[edsiper]

thanks everybody