[Enhancement] add output kafka support for zstd

[gwy1995]

Proposed commit message

add output kafka support for zstd

Checklist

• [x] My code follows the style guidelines of this project
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] I have made corresponding change to the default configuration files
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[cla-checker-service[bot]]

💚 CLA has been signed

[mergify[bot]]

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @gwy1995? 🙏. For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify[bot]]

backport-8.x has been added to help with the transition to the new branch 8.x. If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[pierrehilbert]

Hello @gwy1995 Thanks for your contribution. Could you please sign our CLA?

[gwy1995]

Hello @gwy1995 Thanks for your contribution. Could you please sign our CLA?

I have signed it for three times , but it still shows as unsigned in this issue. What should I do ?

[pierrehilbert]

Did you sign it with the same email than your github account?

[gwy1995]

Did you sign it with the same email than your github account?

yes, I have sign it today and yesterday for more than three times.

[faec]

Hi @gwy1995, we can try figuring out where the CLA checker is having trouble, but another issue I notice is that you removed Elastic's sarama fork and replaced it with an older (downgraded) upstream version. (See the readme in the fork for details.) The fork includes fixes for serious issues, so any changes to the sarama version should either update the fork's base version, or update to a mainline version that includes the fixes we need.

[gwy1995]

Hi @gwy1995, we can try figuring out where the CLA checker is having trouble, but another issue I notice is that you removed Elastic's sarama fork and replaced it with an older (downgraded) upstream version. (See the readme in the fork for details.) The fork includes fixes for serious issues, so any changes to the sarama version should either update the fork's base version, or update to a mainline version that includes the fixes we need.

Thanks for reminding me of this. I have rolled back the sarama version. I found that the elastic version of sarama has merged the changes required by zstd, so just add one line of configuration. I wrote the details in the comments.Please take another look

[leehinman]

looks good. Any chance you could add a test case to https://github.com/elastic/beats/blob/42dc965a50853c282ef4407683711e1cf5b3bbb8/libbeat/outputs/kafka/kafka_integration_test.go#L57 ?

[gwy1995]

looks good. Any chance you could add a test case to

https://github.com/elastic/beats/blob/42dc965a50853c282ef4407683711e1cf5b3bbb8/libbeat/outputs/kafka/kafka_integration_test.go#L57

?

I add a test case. But I don't know why it didn't pass CI. I have tested it in the local environment.

[pierrehilbert]

/test

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/kafka_add_zstd upstream/feature/kafka_add_zstd
git merge upstream/main
git push upstream feature/kafka_add_zstd

[leehinman]

/test

[pierrehilbert]

/test

[pierrehilbert]

@gwy1995 Could you please retry to sign the CLA? This is a mandatory step to be able to merge your PR.

[gwy1995]

@gwy1995 Could you please retry to sign the CLA? This is a mandatory step to be able to merge your PR.

hi, I finally successfully signed the CLA

[pierrehilbert]

/test

[pierrehilbert]

We have a failing test @gwy1995

=== Failed
--
  | === FAIL: libbeat/outputs/kafka TestKafkaPublish/run_test(13):_publish_message_with_zstd_compression_to_test_topic (0.00s)
  | kafka_integration_test.go:278: unknown/unsupported kafka version '2.2.2' accessing 'version'

[gwy1995]

We have a failing test @gwy1995

=== Failed
--
  | === FAIL: libbeat/outputs/kafka TestKafkaPublish/run_test(13):_publish_message_with_zstd_compression_to_test_topic (0.00s)
  | kafka_integration_test.go:278: unknown/unsupported kafka version '2.2.2' accessing 'version'

I have changed version, could you please test again

[pierrehilbert]

/test

[pierrehilbert]

/test

[gwy1995]

We have a failing test @gwy1995

=== Failed
--
  | === FAIL: libbeat/outputs/kafka TestKafkaPublish/run_test(13):_publish_message_with_zstd_compression_to_test_topic (0.00s)
  | kafka_integration_test.go:278: unknown/unsupported kafka version '2.2.2' accessing 'version'

I have changed version, could you please test again

tests are passed?

[pierrehilbert]

run docs-build

[pierrehilbert]

Looks good to go! Thanks for your contribution