confluent-kafka-python icon indicating copy to clipboard operation
confluent-kafka-python copied to clipboard

Published 20 hours ago • verified publisher icon confluentinc

SASL Authentication issue

Open PrabhanjanDesai opened this issue 1 year ago • 4 comments

Description

We have configured strimzi kafka with 9092 as non secured port and with 9093 as secured port. While trying to produce the messages by connecting to 9092 (non secured port) "SASL Authentication error" is thrown intermittently. This error should be thrown while connecting to secure port (9093) when there is an authentication issue.

Error details - Unhandled exception in thread started by <bound method Thread._bootstrap of <Thread(Thread-38, started daemon 139807049099008)>> Traceback (most recent call last): File "/kafka-wrapper/producer_oauth.py", line 87, in produce_to_topic producer.poll(0.1) File "/kafka-wrapper/producer_oauth.py", line 25, in error_callback raise(err) cimpl.KafkaError: KafkaError{code=_TRANSPORT,val=-195,str="kafka-bootstraphost-0.kafka-bootstraphost-brokers.namespace.svc:9092/0: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 1ms in state UP)"} During handling of the above exception, another exception occurred:

How to reproduce

NA

Checklist

Please provide the following information:

  • [ ] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): 2.3.0
  • [ ] Apache Kafka broker version:
  • [ ] Client configuration: {...} -

config = {

  | 'bootstrap.servers': self.broker,   | 'linger.ms': 60,   | 'queue.buffering.max.messages': 5000000,   | 'compression.type': 'lz4',   | 'batch.num.messages': 50000,   | 'batch.size': 200000,   | 'logger': logger,   | 'acks': 0 ,   | 'error_cb': self.error_callback,   | }

  • [ ] Operating system: SUSE linux

  • [ ] Provide client logs (with 'debug': '..' as necessary) - Unhandled exception in thread started by <bound method Thread._bootstrap of <Thread(Thread-38, started daemon 139807049099008)>> Traceback (most recent call last): File "/kafka-wrapper/producer_oauth.py", line 87, in produce_to_topic producer.poll(0.1) File "/kafka-wrapper/producer_oauth.py", line 25, in error_callback raise(err) cimpl.KafkaError: KafkaError{code=_TRANSPORT,val=-195,str="kafka-bootstraphost-0.kafka-bootstraphost-brokers.namespace.svc:9092/0: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 1ms in state UP)"} During handling of the above exception, another exception occurred:

  • [ ] Provide broker log excerpts - Not Available

  • [ ] Critical issue - Not Available

PrabhanjanDesai avatar Mar 04 '24 07:03 PrabhanjanDesai

By secured and non secured, you mean authentication (SASL) or secured communication (SSL)?

What do you mean by intermittent? Does it work properly for some time and then disconnects randomly?

In general, this error happens when the broker listener is configured for SASL authentication but the client is not. Can you please verify this again once?

pranavrth avatar Mar 06 '24 09:03 pranavrth

By secured and non secured, you mean authentication (SASL) or secured communication (SSL)? ****It's MTLS SSL secure communication **** What do you mean by intermittent? Does it work properly for some time and then disconnects randomly? Yes, correct In general, this error happens when the broker listener is configured for SASL authentication but the client is not. Can you please verify this again once? We are connecting to insecure port , below is our kafka configuration : kafka: authorization: superUsers: - ANONYMOUS type: simple config: auto.create.topics.enable: false default.replication.factor: 3 inter.broker.protocol.version: "3.3" min.insync.replicas: 2 offsets.topic.replication.factor: 3 retries: 10 retry.backoff.ms: 2000 transaction.state.log.min.isr: 2 transaction.state.log.replication.factor: 3 listeners: - name: plain port: 9092 tls: false type: internal name: external port: 9094 tls: true type: cluster-ip

PrabhanjanDesai avatar Mar 07 '24 13:03 PrabhanjanDesai

Can you please enable debug logging and provide the generated logs?

pranavrth avatar Mar 13 '24 07:03 pranavrth

Hi guys

I'm having same problem to connect to aws msk sasl_cram,

Producer config I tried with version 2.0.2, 2,2.0 and 2.3.0

    producer_conf = {
        'bootstrap.servers': boostrap_server,
        "client.id": f"efop-sap-integration-admin-{uuid.uuid4().hex}",
        "acks": "all",
        "enable.idempotence": True,
        "security.protocol": "SASL_SSL",
        "sasl.mechanism": "SCRAM-SHA-512",
        "sasl.username": config.sasl_username,
        "sasl.password": config.sasl_password,
    }

    return Producer(producer_conf)

I just remove the server identification form url logs

│ %7|1711465645.879|TOPPAR|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://.amazonaws]: sasl_ssl://amazonaws.com:9096/2: extraction_hubspot_cha │
│ %7|1711465645.879|PRODUCE|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://.amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: extraction_hubspot_ch │
│ %7|1711465645.879|SEND|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://amazonaws]: sasl_ssl://.amazonaws.com:9096/2: Sent partial ProduceRequ │
│ %7|1711465645.879|SEND|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://.amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: Sent partial ProduceRequ │
│ %7|1711465645.879|SEND|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: Sent ProduceRequest (v7, │
│ %7|1711465645.883|RECV|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: Received ProduceResponse │
│ %7|1711465645.883|MSGSET|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: extraction_hubspot_cha │
│ 200 POST /v1/api/driverhubspot/raw/change-tracking (10.42.3.52) 1069.67ms                                                                                                                                                                                                    │
│ %6|1711465647.034|FAIL|webhook-deal-changetracking-d5b58b4caa6c462b9d0985fb9475d2b1#producer-3| [thrd:amazonaws.com:9096/b]: b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/bootstrap: Disconnected while requesti │
│ %7|1711465648.574|PARTITIONER|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:app]: extraction_hubspot_changeset [5] is the new sticky partition                                                                                              │
│ %7|1711465648.574|WAKEUP|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:app]: sasl_ssl://amazonaws.com:9096/2: Wake-up: flushing                                                                  │
│ %7|1711465648.574|TOPPAR|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: extraction_hubspot_cha │
│ %7|1711465648.574|WAKEUP|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:app]: sasl_ssl://amazonaws.com:9096/1: Wake-up: flushing                                                                  │
│ %7|1711465648.574|WAKEUP|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:app]: Wake-up sent to 2 broker threads in state >= UP: flushing

adrian0ribeir0 avatar Mar 26 '24 17:03 adrian0ribeir0