Ceph: support bucket notifications?

[matzew]

The ceph source and sink from kamelet seems to be dealing with objects and not notifications.

We have something like this currently in Knative: https://github.com/knative-extensions/eventing-ceph/blob/main/pkg/adapter/adapter.go#L91

However, with the general move towards eventing-integrations and using Kamelets inside, I was wondering if it is an option to add bucket notifications to the kamelet catalog.

this can do much more than what the current eventing-ceph code does. e.g. configuring the notifications, and maybe integrate with the ceph-source so that objects will be consumed based on the notifications. creating a notification based mechanism instead of polling (as the current ceph-source does) is much more efficient.

[oscerd]

We could do something like the event-based s3: https://github.com/apache/camel-kamelets/blob/main/kamelets/aws-s3-event-based-source.kamelet.yaml

There, we are consuming events from bucket and react to an upload. It should really be the same for Ceph. Does this answer your question?

Thanks

[yuvalif]

yes, it is similar concept, but in case of ceph we ae using kafka not sqs. also in cepoh we have some extensions to the bucket notification configuration that you might want to expose to the conf

[oscerd]

The underline component is using AWS SDK S3 Java client, so unless there is a good reason or something useful on S3 too, we won't add more configuration in the component.

[yuvalif]

The underline component is using AWS SDK S3 Java client, so unless there is a good reason or something useful on S3 too, we won't add more configuration in the component.

to setup bucket notifications and the queue you would need S3 and SQS APIs. but this limits you to AWS, since there is no open source implementation of SQS. in the case of ceph, the queue is provided by kafka, which is available to anyone.