feast icon indicating copy to clipboard operation
feast copied to clipboard

KafkaDataSource to allow for additional Kafka Settings

Open zerafachris opened this issue 9 months ago • 2 comments

Is your feature request related to a problem? Please describe. The current implementation of KafkaSource should also allow for additional kafka settings as per https://kafka.apache.org/documentation.html#consumerconfigs and https://kafka.apache.org/documentation.html#producerconfigs.

For example, in a kafka implementation with SSL Security Protocol, the current kafka stream cannot be used.

Describe the solution you'd like The KafkaOptions (https://rtd.feast.dev/en/master/_modules/feast/data_source.html#KafkaOptions) class should be extended to allow for https://kafka.apache.org/documentation.html#consumerconfigs and https://kafka.apache.org/documentation.html#producerconfigs settings.

As a starting point, maybe focus on security and then move to additional consumering/producing parameterization. Security parameters could be something like: security_protocol, ssl_context, sasl_mechanism, sasl_plain_username, sasl_plain_password. See AIOKafka API (https://aiokafka.readthedocs.io/en/stable/api.html) for a mapping of Kafka settings to consumer and producer apis.

Alternatively, use AIOKafka directly as Feast Kafka engine? [Apologies but could not find the current Kafka engine used by Feast]

Describe alternatives you've considered Nothing yet.

Additional context Not applicable.

zerafachris avatar Jan 05 '25 08:01 zerafachris