Support reading events from Kafka

[hmadison]

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area plugins

What this PR does / why we need it:

This makes an attempt to resolve a line item in #228 around Kafka support. The changes adds a new plugin which allows for a consumer group to read the “bytes off the wire” from a given set of Kafka topics. It explicitly does not support parsing these bytes since Kafka’s internal model is inherently byte based. Instead, we can delegate the parsing to extractor plugins which can build on top of the events provided by this plugin.

Which issue(s) this PR fixes:

In part #228.

Special notes for your reviewer:

• From a test perspective, this makes use of testcontainers/docker to validate the behavior of the plugin. I’m not sure if this work under the current test infrastructure for plugins.
• This seems to have some event time sensitivity issues. At low levels of traffic messages are being read in the go routine but are dropped downstream.

[poiana]

Welcome @hmadison! It looks like this is your first PR to falcosecurity/plugins 🎉

[leogr]

Hey @hmadison

Welcome and thank you for this PR! :)

Just a couple of questions:

• Could you like to register your plugin and pick the next available ID?
• Are you willing to maintain this plugin in the long-term? If yes, I'd recommend proposing yourself as reviewer for this plugin.

cc @falcosecurity/plugins-maintainers

Also, I'm not a Kafka expert, so I'm not likely the right person to review this PR. Please, be patient we will find someone :)

Thank you :pray:

[hmadison]

Hi @leogr,

I'm fine maintaining this long term and helping with review.

Thanks, Hunter

[leogr]

Hi @leogr,

I'm fine maintaining this long term and helping with review.

Thanks, Hunter

This is great! So let's proceed this way:

• We will review and merge this PR (it will take a bit, but we do our best). Cc @Issif @alacuku. May you help with this?
• Once the PR is merged, you will add yourself as a reviewer in a separate PR (and I will sponsor you), then we will do the onboarding

Would this work for you? :)

[Issif]

/assign

[Issif]

Hi,

I'll take a look when free, it can take a week or 2.

[hmadison]

@leogr,

Works great!

Thanks, Hunter

[leogr]

/assign

[leogr]

Also, this test is constantly failing, can you take a look?

cc @alacuku @Andreagit97 @FedeDP. The failure might not be related to this PR. Could you please take a look at it, too? :pray:

[alacuku]

Also, this test is constantly failing, can you take a look?

cc @alacuku @Andreagit97 @FedeDP. The failure might not be related to this PR. Could you please take a look at it, too? 🙏

It seems related to the go version: go: go.mod requires go >= 1.22.2 (running go 1.21.9; GOTOOLCHAIN=local)

@hmadison, could you please bump the go version used in our CI?

[github-actions[bot]]

Rules files suggestions

[leogr]

Hey @hmadison

Unfortunately, we do not allow merge commits in our git history.

Could you rebase this PR and remove the merge commit, pls? :pray:

[github-actions[bot]]

Rules files suggestions

[leogr]

/verify-owners

[github-actions[bot]]

Rules files suggestions

[poiana]

LGTM label has been added.

Git tree hash: 31e71f09208110d64fbe2a612d02e6f4ea7ba19f

[poiana]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hmadison, leogr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~OWNERS~~ [leogr]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[leogr]

Also:

I'm going to release the plugin next week

[leogr]

/hold cancel