eventing icon indicating copy to clipboard operation
eventing copied to clipboard

eks use aws web identity token for SQS queue

Open prabodh1194 opened this issue 10 months ago • 8 comments

Hi,

I have an EKS cluster and I want to create a SQS integration source on this. The docs suggest that I configure AWS secrets for auth purpose. However, my other pods actually use the service account based auth wherein EKS automounts a web identity token on my pod and then all AWS services use those by default.

How can I configure the IntegrationSource in a similar manner?

Thank you.

prabodh1194 avatar Feb 02 '25 03:02 prabodh1194

Hi, there is the option to set

  • CAMEL_KAMELET_AWS_SQS_SOURCE_USE_DEFAULT_CREDENTIALS_PROVIDER=true

in combination with

  • CAMEL_KAMELET_AWS_SQS_SOURCE_USE_SESSION_CREDENTIALS=true

The SQS client should expect to use session credentials then. This is useful in situation in which the user needs to assume an IAM role for doing operations in SQS.

You can then set

  • CAMEL_KAMELET_AWS_SQS_SOURCE_SESSION_TOKEN=<the_sessionToken>

via secretRefValue for instance.

Is this something you are looking for?

christophd avatar Feb 10 '25 13:02 christophd

hmm. looking good. i'll need to try using this.

prabodh1194 avatar Feb 10 '25 14:02 prabodh1194

How do I set CAMEL_KAMELET_AWS_SQS_SOURCE_SESSION_TOKEN ? Do you mean inject it ? This is very unclear . Where does one put these env variable ?

It will be great if there is an example on how to do this in doc.

cloudbow avatar Mar 13 '25 17:03 cloudbow

so i've tried this with ContainerSource using AWS Pod Identity I've setup the ServiceAccount to have a role with SQS permissions, and can't see how this is not working.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: aws-sqs
  namespace: testing

---
apiVersion: sources.knative.dev/v1
kind: ContainerSource
metadata:
  annotations:
    sources.knative.dev/creator: system:serviceaccount:knative-eventing:eventing-controller
    sources.knative.dev/lastModifier: system:serviceaccount:knative-eventing:eventing-controller
  labels:
    app.kubernetes.io/name: aws-sqs-testing-received-source
  name: aws-sqs-testing-received-source-containersource
  namespace: testing
spec:
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: testing-broker
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/name: testing-received-source
    spec:
      serviceAccountName: "aws-sqs"
      containers:
      - env:
        - name: CAMEL_KNATIVE_CLIENT_SSL_ENABLED
          value: "true"
        - name: CAMEL_KNATIVE_CLIENT_SSL_CERT_PATH
          value: /knative-custom-certs/knative-eventing-bundle.pem
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_REGION
          value: us-east-2
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_OVERRIDEENDPOINT
          value: "false"
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_QUEUENAMEORARN
          value: testing-received
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_DELETEAFTERREAD
          value: "true"
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_AUTOCREATEQUEUE
          value: "false"
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_AMAZONAWSHOST
          value: amazonaws.com
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_PROTOCOL
          value: https
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_GREEDY
          value: "false"
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_DELAY
          value: "500"
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_MAXMESSAGESPERPOLL
          value: "1"
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_WAITTIMESECONDS
          value: "0"
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_VISIBILITYTIMEOUT
          value: "0"
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_USE_DEFAULT_CREDENTIALS_PROVIDER
          value: "true"
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_USE_SESSION_CREDENTIALS
          value: "true"
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_SESSION_TOKEN
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: CAMEL_KAMELET_AWS_SQS_SOURCE_QUEUE_U_R_L
          value: https://sqs.us-east-2.amazonaws.com/XXXXXXXXXXXX/testing-received
        image: gcr.io/knative-nightly/aws-sqs-source:v20250424-ec8d187
        imagePullPolicy: IfNotPresent
        name: source
        resources: {}
        volumeMounts:
        - mountPath: "/var/run/secrets/eks.amazonaws.com/serviceaccount/"
          name: aws-token
      volumes:
      - name: aws-token
        projected:
          sources:
          - serviceAccountToken:
              audience: "sts.amazonaws.com"
              expirationSeconds: 86400
              path: token

still receiving messages about

Caused by: java.lang.IllegalArgumentException: useDefaultCredentialsProvider is set to false, useProfileCredentialsProvider is set to false, useSessionCredentials is set to false, AmazonSQSClient or accessKey and secretKey must be specified
    at org.apache.camel.component.aws2.sqs.Sqs2Component.createEndpoint(Sqs2Component.java:71)
    at org.apache.camel.support.DefaultComponent.createEndpoint(DefaultComponent.java:171)
    at org.apache.camel.impl.engine.AbstractCamelContext.doGetEndpoint(AbstractCamelContext.java:807)
    ... 34 more

So, for some reason, the env-vars are not being used in the pod, even though it shows in the podSpec

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2025-04-24T19:30:09Z"
  generateName: aws-sqs-testing-received-source-containersource-deployment-84445f74c6-
  labels:
    app.kubernetes.io/name: aws-sqs-testing-received-source
    pod-template-hash: 84445f74c6
    sources.knative.dev/containerSource: aws-sqs-testing-received-source-containersource
    sources.knative.dev/source: container-source-controller
  name: aws-sqs-testing-received-source-containersource-deplo46f4s
  namespace: testing
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: aws-sqs-testing-received-source-containersource-deployment-84445f74c6
    uid: 6d9736cc-779b-47bd-822e-2957cb6430c3
  resourceVersion: "705275331"
  uid: 1c2090ba-fafe-49b0-b0b4-245cc535e10e
spec:
  containers:
  - env:
    - name: CAMEL_KNATIVE_CLIENT_SSL_ENABLED
      value: "true"
    - name: CAMEL_KNATIVE_CLIENT_SSL_CERT_PATH
      value: /knative-custom-certs/knative-eventing-bundle.pem
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_REGION
      value: us-east-2
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_OVERRIDEENDPOINT
      value: "false"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_QUEUENAMEORARN
      value: testing-received
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_DELETEAFTERREAD
      value: "true"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_AUTOCREATEQUEUE
      value: "false"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_AMAZONAWSHOST
      value: amazonaws.com
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_PROTOCOL
      value: https
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_GREEDY
      value: "false"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_DELAY
      value: "500"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_MAXMESSAGESPERPOLL
      value: "1"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_WAITTIMESECONDS
      value: "0"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_VISIBILITYTIMEOUT
      value: "0"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_USE_DEFAULT_CREDENTIALS_PROVIDER
      value: "true"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_USE_SESSION_CREDENTIALS
      value: "true"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_SESSION_TOKEN
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_QUEUE_U_R_L
      value: https://sqs.us-east-2.amazonaws.com/XXXXXXXXXXXX/testing-received
    - name: K_SINK
      value: http://broker-ingress.knative-eventing.svc.cluster.local/testing/testing-broker
    - name: K_CE_OVERRIDES
    image: gcr.io/knative-nightly/aws-sqs-source:v20250424-ec8d187
    imagePullPolicy: IfNotPresent
    name: source
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount/
      name: aws-token
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-z7snt
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-10-0-129-13.us-east-2.compute.internal
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: aws-sqs
  serviceAccountName: aws-sqs
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: aws-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token
  - name: kube-api-access-z7snt
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-04-24T19:30:10Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2025-04-24T19:30:09Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-04-24T19:51:37Z"
    message: 'containers with unready status: [source]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-04-24T19:51:37Z"
    message: 'containers with unready status: [source]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-04-24T19:30:09Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://ccf6277581461978d9f7a8bde1a04d542d82a2a803191935056ac00bcc59b48f
    image: gcr.io/knative-nightly/aws-sqs-source:v20250424-ec8d187
    imageID: gcr.io/knative-nightly/aws-sqs-source@sha256: 9439811d4cb007821d51e46ec3b8e98917a8c181a63d386d86d2d7928f739eea
    lastState:
      terminated:
        containerID: containerd://ccf6277581461978d9f7a8bde1a04d542d82a2a803191935056ac00bcc59b48f
        exitCode: 1
        finishedAt: "2025-04-24T19:51:36Z"
        reason: Error
        startedAt: "2025-04-24T19:51:35Z"
    name: source
    ready: false
    restartCount: 9
    started: false
    state:
      waiting:
        message: back-off 5m0s restarting failed container=source pod=aws-sqs-testing-received-source-containersource-deplo46f4s_testing(1c2090ba-fafe-49b0-b0b4-245cc535e10e)
        reason: CrashLoopBackOff
    volumeMounts:
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount/
      name: aws-token
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-z7snt
      readOnly: true
      recursiveReadOnly: Disabled
  hostIP: 10.0.129.13
  hostIPs:
  - ip: 10.0.129.13
  phase: Running
  podIP: 10.0.139.59
  podIPs:
  - ip: 10.0.139.59
  qosClass: BestEffort
  startTime: "2025-04-24T19:30:09Z"
  • The contents have been scrubbed for privacy, but everything else remains intact. -

liskl avatar Apr 24 '25 20:04 liskl

How do I set CAMEL_KAMELET_AWS_SQS_SOURCE_SESSION_TOKEN ? Do you mean inject it ? This is very unclear . Where does one put these env variable ?

It will be great if there is an example on how to do this in doc.

@matzew @christophd can those be set in the IntegrationSource CR, or is it required to go via the ContainerSource?

creydr avatar May 08 '25 13:05 creydr

@matzew @christophd Currently its mandatory to pass auth under spec.aws according to documenation. How to use DEFAULT_CREDENTIALS_PROVIDER in IntegrationSource? Can you provide the example?

prashanthm10 avatar May 16 '25 08:05 prashanthm10

@prashanthm10 @matzew @christophd it seems there currently is no way to use IntegrationSource with the DEFAULT_CREDENTIALS_PROVIDER or IRSA, or using AWS PodIdentities, cause it requires presently the spec.aws field, hence my example above replicating what the IntegrationSource creates, but using the kamelet ENVARS for enabling the default provider, but it looks like the image we are using gcr.io/knative-nightly/aws-sqs-source@sha256:9439811d4cb007821d51e46ec3b8e98917a8c181a63d386d86d2d7928f739eea somehow overrides the values I tried to set

    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_USE_DEFAULT_CREDENTIALS_PROVIDER
      value: "true"
    - name: CAMEL_KAMELET_AWS_SQS_SOURCE_USE_SESSION_CREDENTIALS
      value: "true"

and we still get the error

Caused by: java.lang.IllegalArgumentException: useDefaultCredentialsProvider is set to false, useProfileCredentialsProvider is set to false, useSessionCredentials is set to false, AmazonSQSClient or accessKey and secretKey must be specified
    at org.apache.camel.component.aws2.sqs.Sqs2Component.createEndpoint(Sqs2Component.java:71)
    at org.apache.camel.support.DefaultComponent.createEndpoint(DefaultComponent.java:171)
    at org.apache.camel.impl.engine.AbstractCamelContext.doGetEndpoint(AbstractCamelContext.java:807)
    ... 34 more

see issuecomment-2828734485

liskl avatar May 18 '25 11:05 liskl

/triage accepted

matzew avatar May 22 '25 08:05 matzew

Do we have any estimate on when could this go to production?

prashanthm10 avatar Jun 28 '25 05:06 prashanthm10

@liskl did you ever sort out why using the ContainerSource directly didn't work?

dprotaso avatar Sep 12 '25 13:09 dprotaso

Nope never did get this working in v1.17, I can try again shortly as I just updated to v1.18 and see where we are currently. But I don't think anything has changed in that section of the codebase yet

liskl avatar Sep 12 '25 14:09 liskl

@christophd do you know why the use of the image with the CAMEL_ env vars wouldn't work as a workaround?

dprotaso avatar Sep 12 '25 14:09 dprotaso

I need to find the time to reproduce this. I do not see any reason why CAMEL_ env vars shouldn't work with ContainerSource.

@matzew is the gcr.io/knative-nightly image up to date with the latest versions? I think the Camel Kamelets have received some updates and fixes regarding AWS IAM connectivity recently, so we have to make sure that the images are using these updated versions of Camel.

Also, in the example from @liskl I see the following:

- name: CAMEL_KAMELET_AWS_SQS_SOURCE_SESSION_TOKEN
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name

Not sure about this. How is this supposed to work?

Another thing about CAMEL_KAMELET_AWS_SQS_SOURCE_USE_DEFAULT_CREDENTIALS_PROVIDER and CAMEL_KAMELET_AWS_SQS_SOURCE_USE_SESSION_CREDENTIALS: This is exclusive one or the other.

AFAIU if CAMEL_KAMELET_AWS_SQS_SOURCE_USE_DEFAULT_CREDENTIALS_PROVIDER is set there is no need to also enable CAMEL_KAMELET_AWS_SQS_SOURCE_USE_SESSION_CREDENTIALS

But as mentioned I need to find the time to reproduce this myself. Thanks!

christophd avatar Sep 12 '25 14:09 christophd

I think the Camel Kamelets have received some updates and fixes regarding AWS IAM connectivity recently, so we have to make sure that the images are using these updated versions of Camel.

@christophd - What version of Camel has the changes you're talking about? Knative 1.19 has Camel 4.12.0 and 1.18 has Camel 4.9.0

dprotaso avatar Sep 16 '25 17:09 dprotaso

@christophd @matzew I had someone test out the latest eventing-integration nightly images and CAMEL_KAMELET_AWS_SQS_SOURCE_USE_DEFAULT_CREDENTIALS_PROVIDER worked.

dprotaso avatar Sep 16 '25 20:09 dprotaso

I need to find the time to reproduce this. I do not see any reason why CAMEL_ env vars shouldn't work with ContainerSource.

@matzew is the gcr.io/knative-nightly image up to date with the latest versions? I think the Camel Kamelets have received some updates and fixes regarding AWS IAM connectivity recently, so we have to make sure that the images are using these updated versions of Camel.

Also, in the example from @liskl I see the following:

  • name: CAMEL_KAMELET_AWS_SQS_SOURCE_SESSION_TOKEN valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name Not sure about this. How is this supposed to work?

Another thing about CAMEL_KAMELET_AWS_SQS_SOURCE_USE_DEFAULT_CREDENTIALS_PROVIDER and CAMEL_KAMELET_AWS_SQS_SOURCE_USE_SESSION_CREDENTIALS: This is exclusive one or the other.

AFAIU if CAMEL_KAMELET_AWS_SQS_SOURCE_USE_DEFAULT_CREDENTIALS_PROVIDER is set there is no need to also enable CAMEL_KAMELET_AWS_SQS_SOURCE_USE_SESSION_CREDENTIALS

But as mentioned I need to find the time to reproduce this myself. Thanks!

that was supposed to make the session name the knative name, but my incompetence didn't catch that that was actually the TOKEN not the session name

forgive my stupidity on that one. @matzew

liskl avatar Sep 16 '25 21:09 liskl

@christophd @matzew I had someone test out the latest eventing-integration nightly images and CAMEL_KAMELET_AWS_SQS_SOURCE_USE_DEFAULT_CREDENTIALS_PROVIDER worked.

Could you show me an example of this? I'll be trying this out next week to see if I can utilize the Knative functions again with EKS v1.33 and pod identities, now that we have moved away from IRSA roles.

liskl avatar Sep 16 '25 21:09 liskl

@liskl nightly images are here: https://gcsweb.knative.dev/gcs/knative-nightly/eventing-integrations/latest/

dprotaso avatar Sep 16 '25 22:09 dprotaso

@liskl Here is a sample ContainerSource that is using the aws-sqs-source nightly image and the Camel useDefaultCredentialsProvider=true property. We are using IRSA so you'll notice that the Deployment template includes a serviceAccountName so that the pods will be assigned the correct IAM role. I am working on a PR to make this available through the AWS IntegrationSource and IntegrationSink resources but in the meantime you should be able to directly create a ContainerSource.

{
    "apiVersion": "sources.knative.dev/v1",
    "kind": "ContainerSource",
    "metadata": {
        "annotations": {
            "sources.knative.dev/creator": "system:serviceaccount:knative-eventing:eventing-controller",
            "sources.knative.dev/lastModifier": "system:serviceaccount:knative-eventing:eventing-controller"
        },
        "creationTimestamp": "2025-09-16T18:03:47Z",
        "generation": 4,
        "labels": {
            "app.kubernetes.io/name": "integration-source-aws-sqs"
        },
        "name": "integration-source-aws-sqs-containersource",
        "namespace": "knative-demo",
        "ownerReferences": [
            {
                "apiVersion": "sources.knative.dev/v1alpha1",
                "blockOwnerDeletion": true,
                "controller": true,
                "kind": "IntegrationSource",
                "name": "integration-source-aws-sqs",
                "uid": "297ad4e6-6c37-40a4-9e0d-2ca97d4b2fda"
            }
        ],
        "resourceVersion": "14790915",
        "uid": "6880c041-277a-4bf2-8c15-c22ccb8722e7"
    },
    "spec": {
        "sink": {
            "ref": {
                "apiVersion": "eventing.knative.dev/v1",
                "kind": "Broker",
                "name": "bookstore-broker"
            }
        },
        "template": {
            "metadata": {
                "creationTimestamp": null,
                "labels": {
                    "app.kubernetes.io/name": "integration-source-aws-sqs"
                }
            },
            "spec": {
                "containers": [
                    {
                        "env": [
                            {
                                "name": "CAMEL_KNATIVE_CLIENT_SSL_ENABLED",
                                "value": "true"
                            },
                            {
                                "name": "CAMEL_KNATIVE_CLIENT_SSL_CERT_PATH",
                                "value": "/knative-custom-certs/knative-eventing-bundle.pem"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_REGION",
                                "value": "us-east-1"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_OVERRIDEENDPOINT",
                                "value": "false"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_QUEUE_NAME_OR_ARN",
                                "value": "arn:aws:sqs:us-east-1:123456789012:knative-integration-source.fifo"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_DELETEAFTERREAD",
                                "value": "true"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_AUTOCREATEQUEUE",
                                "value": "false"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_AMAZONAWSHOST",
                                "value": "amazonaws.com"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_PROTOCOL",
                                "value": "https"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_GREEDY",
                                "value": "false"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_DELAY",
                                "value": "500"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_MAXMESSAGESPERPOLL",
                                "value": "1"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_WAITTIMESECONDS",
                                "value": "0"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_VISIBILITYTIMEOUT",
                                "value": "0"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_USEDEFAULTCREDENTIALSPROVIDER",
                                "value": "true"
                            }
                        ],
                        "image": "gcr.io/knative-nightly/aws-sqs-source@sha256:660ae70a4424613e7d8795bedbff9f2099d75d1498ba7b3590e622f894b1753c",
                        "imagePullPolicy": "IfNotPresent",
                        "name": "source",
                        "resources": {}
                    }
                ],
                "serviceAccountName": "integration-source-aws-sqs"
            }
        }
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2025-09-16T22:46:02Z",
                "status": "True",
                "type": "Ready"
            },
            {
                "lastTransitionTime": "2025-09-16T22:46:02Z",
                "status": "True",
                "type": "ReceiveAdapterReady"
            },
            {
                "lastTransitionTime": "2025-09-16T18:03:47Z",
                "status": "True",
                "type": "SinkBindingReady"
            }
        ],
        "observedGeneration": 4,
        "sinkUri": "http://kafka-broker-ingress.knative-eventing.svc.cluster.local/knative-demo/bookstore-broker"
    }
}

This is the policy attached to the IRSA role for the ServiceAccount:

    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "sts:GetCallerIdentity",
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "sqs:SendMessageBatch",
                "sqs:SendMessage",
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes",
                "sqs:DeleteMessage",
                "sqs:ChangeMessageVisibility"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:us-east-1:123456789012:knative-integration-source.fifo"
        }
    ]
}

qswinson avatar Sep 16 '25 22:09 qswinson

@liskl Here is a sample ContainerSource that is using the aws-sqs-source nightly image and the Camel useDefaultCredentialsProvider=true property. We are using IRSA so you'll notice that the Deployment template includes a serviceAccountName so that the pods will be assigned the correct IAM role. I am working on a PR to make this available through the AWS IntegrationSource and IntegrationSink resources but in the meantime you should be able to directly create a ContainerSource.

{
    "apiVersion": "sources.knative.dev/v1",
    "kind": "ContainerSource",
    "metadata": {
        "annotations": {
            "sources.knative.dev/creator": "system:serviceaccount:knative-eventing:eventing-controller",
            "sources.knative.dev/lastModifier": "system:serviceaccount:knative-eventing:eventing-controller"
        },
        "creationTimestamp": "2025-09-16T18:03:47Z",
        "generation": 4,
        "labels": {
            "app.kubernetes.io/name": "integration-source-aws-sqs"
        },
        "name": "integration-source-aws-sqs-containersource",
        "namespace": "knative-demo",
        "ownerReferences": [
            {
                "apiVersion": "sources.knative.dev/v1alpha1",
                "blockOwnerDeletion": true,
                "controller": true,
                "kind": "IntegrationSource",
                "name": "integration-source-aws-sqs",
                "uid": "297ad4e6-6c37-40a4-9e0d-2ca97d4b2fda"
            }
        ],
        "resourceVersion": "14790915",
        "uid": "6880c041-277a-4bf2-8c15-c22ccb8722e7"
    },
    "spec": {
        "sink": {
            "ref": {
                "apiVersion": "eventing.knative.dev/v1",
                "kind": "Broker",
                "name": "bookstore-broker"
            }
        },
        "template": {
            "metadata": {
                "creationTimestamp": null,
                "labels": {
                    "app.kubernetes.io/name": "integration-source-aws-sqs"
                }
            },
            "spec": {
                "containers": [
                    {
                        "env": [
                            {
                                "name": "CAMEL_KNATIVE_CLIENT_SSL_ENABLED",
                                "value": "true"
                            },
                            {
                                "name": "CAMEL_KNATIVE_CLIENT_SSL_CERT_PATH",
                                "value": "/knative-custom-certs/knative-eventing-bundle.pem"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_REGION",
                                "value": "us-east-1"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_OVERRIDEENDPOINT",
                                "value": "false"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_QUEUE_NAME_OR_ARN",
                                "value": "arn:aws:sqs:us-east-1:123456789012:knative-integration-source.fifo"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_DELETEAFTERREAD",
                                "value": "true"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_AUTOCREATEQUEUE",
                                "value": "false"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_AMAZONAWSHOST",
                                "value": "amazonaws.com"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_PROTOCOL",
                                "value": "https"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_GREEDY",
                                "value": "false"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_DELAY",
                                "value": "500"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_MAXMESSAGESPERPOLL",
                                "value": "1"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_WAITTIMESECONDS",
                                "value": "0"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_VISIBILITYTIMEOUT",
                                "value": "0"
                            },
                            {
                                "name": "CAMEL_KAMELET_AWS_SQS_SOURCE_USEDEFAULTCREDENTIALSPROVIDER",
                                "value": "true"
                            }
                        ],
                        "image": "gcr.io/knative-nightly/aws-sqs-source@sha256:660ae70a4424613e7d8795bedbff9f2099d75d1498ba7b3590e622f894b1753c",
                        "imagePullPolicy": "IfNotPresent",
                        "name": "source",
                        "resources": {}
                    }
                ],
                "serviceAccountName": "integration-source-aws-sqs"
            }
        }
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2025-09-16T22:46:02Z",
                "status": "True",
                "type": "Ready"
            },
            {
                "lastTransitionTime": "2025-09-16T22:46:02Z",
                "status": "True",
                "type": "ReceiveAdapterReady"
            },
            {
                "lastTransitionTime": "2025-09-16T18:03:47Z",
                "status": "True",
                "type": "SinkBindingReady"
            }
        ],
        "observedGeneration": 4,
        "sinkUri": "http://kafka-broker-ingress.knative-eventing.svc.cluster.local/knative-demo/bookstore-broker"
    }
}

This is the policy attached to the IRSA role for the ServiceAccount:

    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "sts:GetCallerIdentity",
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "sqs:SendMessageBatch",
                "sqs:SendMessage",
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes",
                "sqs:DeleteMessage",
                "sqs:ChangeMessageVisibility"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:us-east-1:123456789012:knative-integration-source.fifo"
        }
    ]
}

Your amazing @qswinson

liskl avatar Sep 17 '25 02:09 liskl