Connecting to Kafka using Kafka / Confluent REST Proxy

[metaversed]

Feature Request

As an application, it needs to be able to send data to kafka brokers using a REST endpoint that can be configured in one of the plugins that is used by telegraf agent which can handle this use case.

Proposal:

At the moment , we have https://github.com/influxdata/telegraf/tree/master/plugins/outputs/http where this can be handled if possible.

Current behavior:

Not Available

Desired behavior:

Should be able to send metrics data that can be unstructured and should not have a need to enforce schema registration Does it really make sense to expect this feature from a telgraf point of view ?

Use case:

With this, it will be able to manage the requests being sent to Kafka brokers by exposing the HTTP endpoint using an API manager where you could throttle the limits that can be shared by a specific consumer and also identify , secure using API keys or openid. At the moment there is no way to perform event managed while publishing events to an event streaming platform such as Kafka

[danielnelson]

API reference: https://docs.confluent.io/current/kafka-rest/api.html#

[metaversed]

@danielnelson Any ideas if the latest updates from Kafka helps ?

Schema Registry has been made extensible when it comes to new formats, and now users can even start adding their own custom formats to Schema Registry.

The new Protobuf and JSON Schema formats are supported in Schema Registry in addition to all other components of Confluent Platform 5.5.

[danielnelson]

Unfortunately I'm unlikely to have time available to look into this myself anytime soon. I'll try to review if someone would like to work on a prototype.

[sspaink]

@metaversed are you still interested in seeing this feature request implemented? If I understand correctly you want the Kafka output plugin to support the Confleunt API so that you can embed schemas into the requests?

[telegraf-tiger[bot]]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!