Azure event hub with kafka to get unprocessed events.

[gudipudipradeep]

Proposal

consumer application using <"Confluent.Kafka" Version="2.2.0" /> library for cost reduction and some more reasons from application team. As devops engineer what other option we have to read unprocessed events. does Apache kafka keda support Azure event hub..? Please share your thoughts with some sample on it.

Use-Case

Our consumer application using Confluent.Kafka. to get the unprocessed event we need the blog with partition file. but Confluent.Kafka will taken care about the partition and offsets.

does apache kafka keda api support Azure event hub.. if yes please share some guidelines on it

Is this a feature you are interested in implementing yourself?

No

Anything else?

No response

[JorTurFer]

Hello, I'm not sure because I haven't testes it. I think that it works as Azure EventHub has Kafka API compatibility mode, so it should work. You can give a try and share your experience. You also have a Specific Azure EvenHub scaler if you need it

[gudipudipradeep]

@JorTurFer Specific Azure EvenHub scaler is not working because we need partition file in the blob storage. if we work with kafka api no need to maintain partition file in the blob storage.

[gudipudipradeep]

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: iothubinternal-eventhub-scaledobject
  namespace: evt
spec:
  scaleTargetRef:
    name: opintel-common-event-listener-iothubinternal-v1
  cooldownPeriod: 300 #Time in seconds after which Pods should be scaled down to minReplicaCount
  pollingInterval: 30 #Time in seconds after which HPA should check for unprocessed event in the Event Hub
  minReplicaCount: 1
  maxReplicaCount: 6
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: devcaidcdeviceeventspipeline.servicebus.windows.net:9093
        eventHubName: opintel-evt-iot-hub #Internal Event Hub of IOTHub 
        consumerGroup: evt
        topic: opintel-evt-iot-hub
        lagThreshold: "10"
        offsetResetPolicy: latest
        tls: enable
        sasl: plaintext
      authenticationRef:
        name: keda-trigger-auth-kafka-credential
---
apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-secrets
  namespace: default
data:
  authMode: "org.apache.kafka.common.security.plain.PlainLoginModule"
  username: "$ConnectionString"
  password: "password"
---
apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-credential
  namespace: default
spec:
  secretTargetRef:
  - parameter: authMode
    name: keda-kafka-secrets
    key: authMode
  - parameter: username
    name: keda-kafka-secrets
    key: username
  - parameter: password
    name: keda-kafka-secrets
    key: password

above steps I have done for kafka with azure event hub. still getting failures.

[JorTurFer]

What failures do you have? Using EventHub Scaler, have you checked the legacy checkpointer (which doesn't use storage) https://keda.sh/docs/2.12/scalers/azure-event-hub/#checkpointing-behaviour?

[gudipudipradeep]

some time its in Ready state is True or some time its in False fluctuating.

[gudipudipradeep]

keda-operator-7f4cbd5d8b.log keda-operator-67665bbbf5.log

always Unkown

2023-12-18T07:50:45Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "evt", "name": "iothubinternal-eventhub-scaledobject", "error": "error describing topics: kafka server: Request was for a topic or partition that does not exist on this broker"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTopicPartitions
	/workspace/pkg/scalers/kafka_scaler.go:452
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:633
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).GetMetricsAndActivity
	/workspace/pkg/scalers/kafka_scaler.go:620
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricsAndActivityForScaler
	/workspace/pkg/scaling/cache/scalers_cache.go:130
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState
	/workspace/pkg/scaling/scale_handler.go:572
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:236
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:175

[JorTurFer]

Honestly, I have no idea if Kafka scaler can be used for Azure EventHub... @v-shenoy @tomkerkhove , do you have any idea?

[tomkerkhove]

Yes you can, but I believe it's using the Kafka protocol so not sure what scaler to use to be honest :)

https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-kafka-overview

[gudipudipradeep]

@tomkerkhove As per above documentation I setup the kafka keda deployment. but not working. shared logs along with deployment file. Please can you help on this.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to inactivity.