oauth2.0 scope not supported for org.apache.camel.kafkaconnector.awssqssink.CamelAwssqssinkSinkConnector

[L040639]

Caused by: javax.security.auth.login.LoginException: io.strimzi.kafka.oauth.common.HttpException: POST request to https://login.microsoftonline.com/18a59a81-eea8-4c30-948a-d8824cdc2580/oauth2/v2.0/token failed with status 400: {"error":"invalid_request","error_description":"AADSTS90014: The required field 'scope' is missing from the credential. Ensure that you have all the necessary parameters for the login request. Trace ID: afeb7f73-4348-4f19-a398-ac0df5bb2100 Correlation ID: 89c056d2-d633-4813-b9f7-b5ac2b342fac Timestamp: 2025-01-26 09:11:56Z","error_codes":[90014],"timestamp":"2025-01-26 09:11:56Z","trace_id":"afeb7f73-4348-4f19-a398-ac0df5bb2100","correlation_id":"89c056d2-d633-4813-b9f7-b5ac2b342fac","error_uri":"https://login.microsoftonline.com/error?code=90014"}

i am trying to deploy a kafka connector which will authenticate to the oauth protected kafka cluster, so using camel sqs sink connector to sink the data from the topic while using but i am getting above error , does camel sink connector not supporting oauth2.0 scope, below are the configuration files .

Docker file for the kafkaconnect custom image :

FROM strimzi/kafka:0.17.0-kafka-2.4.0

USER root:root

# Create the plugins directory
RUN mkdir -p /opt/kafka/plugins

# Copy the Camel Kafka Connector package to the plugins directory
COPY ./plugins/  /opt/kafka/plugins/

USER 1001

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata: 
  name: my-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.7.0
  replicas: 1
  image: xxxxxx
  bootstrapServers: xxxxx
  tls:
    trustedCertificates:
      - secretName: lk-ca-root-bundle
        certificate: ca.crt
  logging:
    type: inline
    loggers:
      log4j.rootLogger: INFO
  readinessProbe: # (13)
    initialDelaySeconds: 15
    timeoutSeconds: 5
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  config:
    group.id: xxxxxx
    offset.storage.topic: connectoffsets
    config.storage.topic: connectconfigs
    status.storage.topic: connectstatus
    # key.converter: org.apache.kafka.connect.json.JsonConverter
    # value.converter: org.apache.kafka.connect.json.JsonConverter
    # key.converter.schemas.enable: true
    # value.converter.schemas.enable: true
    config.storage.replication.factor: 3
    offset.storage.replication.factor: 3
    status.storage.replication.factor: 3
  template:
    pod:
      metadata:
        annotations:
          eks.amazonaws.com/role-arn: xxxxxx
      imagePullSecrets:
        - name: kafka-artifactory
      terminationGracePeriodSeconds: 120
    serviceAccount:
      metadata:
        annotations:
          eks.amazonaws.com/role-arn: xxxxxxxx
  authentication:
    type: oauth
    clientId: xxxxxxx
    clientSecret:
      key: secret
      secretName: kafka-clientsecret
    scope: xxxxxxx
    tokenEndpointUri: 'https://login.microsoftonline.com/xxxxxxxx

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: my-source-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.camel.kafkaconnector.awssqssink.CamelAwssqssinkSinkConnector
  config:
    topics: "xxxxxx"
    camel.kamelet.aws-sqs-sink.queueNameOrArn: xxxxxxx
    camel.kamelet.aws-sqs-sink.useSessionCredentials: "true"
    camel.kamelet.aws-sqs-sink.region: "us-east-2"

@oscerd any idea ???

[oscerd]

This is eventually something related to strimzi. I don't think it's something on the Ckc side.

[oscerd]

Also is this azure Kafka?

[L040639]

i have deployed this in my openshift cluster, now I have changed my configuration, getting a differnt below error , eventhough i have added the aws-sdk sts plugins still getting below error does camel connectors not handling this ?? @oscerd

To use web identity tokens, the 'sts' service module must be on the class path. (software.amazon.awssdk.auth.credentials.internal.WebIdentityCredentialsUtils) [task-thread-my-source-connector-0]

Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.7.0
  replicas: 1
  bootstrapServers: <xxxxxx>
  tls:
    trustedCertificates:
      - secretName: lk-ca-root-bundle
        certificate: ca.crt
  logging:
    type: inline
    loggers:
      log4j.rootLogger: INFO
  readinessProbe: # (13)
    initialDelaySeconds: 15
    timeoutSeconds: 5
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  config:
    group.id: <xxxxxx>
    offset.storage.topic: connectoffsets
    config.storage.topic: connectconfigs
    status.storage.topic: connectstatus
    config.storage.replication.factor: 3
    offset.storage.replication.factor: 3
    status.storage.replication.factor: 3
    ##newly added
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter.schemas.enable: false
    key.converter.schemas.enable: false
  build:
    output:
      type: docker
      image: <xxxxxx>/<xxxx>
      pushSecret: kafka-artifactory
    plugins:
      - name: camel-aws-sqs-connector
        artifacts:
          - type: tgz
            url: https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws-sqs-sink-kafka-connector/4.8.0/camel-aws-sqs-sink-kafka-connector-4.8.0-package.tar.gz
            sha512sum: adb448cf78af3144a473ea8dc2bfcf5f12fd2531f622125ce58b52e64549e64677d10adab0f338565f4948885776ec5bcc0b0f2ff9d57411e527222e57f189c1
      - name: aws-sdk-sts
        artifacts:
          - type: jar
            url: https://repo1.maven.org/maven2/software/amazon/awssdk/sts/2.30.7/sts-2.30.7-javadoc.jar
            sha512sum: aad89de3d80cca0cc89411268fbd495adcd3b527ba109534725e19e6bf8f9850768fbd086ee159fa431bc684402a67a2a8de8a521ed06edf96f9c5f3b0c4d9ad
  template:
    pod:
      metadata:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::<xxxxx>:role/<xxxx>
      imagePullSecrets:
        - name: kafka-artifactory
      terminationGracePeriodSeconds: 120
    serviceAccount:
      metadata:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::<xxxx>:role/<xxxx>
  authentication:
    type: oauth
    clientId: <xxxx>
    clientSecret:
      key: secret
      secretName: kafka-clientsecret
    scope: 'api://<xxxx>/.default'
    tokenEndpointUri: 'https://login.microsoftonline.com/<xxxx>/oauth2/v2.0/token'
---


apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: my-source-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.camel.kafkaconnector.awssqssink.CamelAwssqssinkSinkConnector
  config:
    topics: "testtopic"
    camel.kamelet.aws-sqs-sink.queueNameOrArn: "arn:aws:sqs:us-east-2:<xxxxx>:<xxxx>"
    camel.kamelet.aws-sqs-sink.useDefaultCredentialsProvider: "true"
    camel.kamelet.aws-sqs-sink.region: "us-east-2"
    consumer.override.group.id: "<xxxxxx>"

[oscerd]

The sts jar should in the connector package, not separated.

[L040639]

tried but seems it is copying in a separate folder, is there a ideal way to keep both in a same file. @oscerd

[oscerd]

No, you need to have the STS JAR in the same folder.