Need some clarification on camel-aws-s3-source-kafka-connector

[stuartm21]

Hello @oscerd ,

We are planning to use the following connect and connector for restoring the topic data from s3 to kafka.

kafka-connect.yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: strimzi-kafka
  namespace: kafka
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.1.0
  replicas: 1
  image: dockerhub/camel-aws-s3-source-kafka-connector:v1.0.0
  bootstrapServers: kafka-bootstrap.myhost.com:443
  externalConfiguration:
    volumes:
      - name: aws-credentials
        secret:
          secretName: aws-credentials
  tls:
    trustedCertificates:
      - secretName: strimzi-cluster-ca-cert
        certificate: ca.crt
  authentication:
    type: tls
    certificateAndKey:
      certificate: user.crt
      key: user.key
      secretName: kafka-user
  config:
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1

source-connector.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: s3-source-connector
  namespace: kafka
  labels:
    strimzi.io/cluster: "strimzi-kafka"
spec:
  class: org.apache.camel.kafkaconnector.awss3source.CamelAwss3sourceSourceConnector
  tasksMax: 2
  config:
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    topics: kafka-topic
    camel.kamelet.aws-s3-source.bucketNameOrArn: test-kafka-connect
    camel.kamlet.aws-s3.configuration.autocloseBody: false
    camel.kamelet.aws-s3-source.accessKey: ${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_access_key_id}
    camel.kamelet.aws-s3-source.secretKey: ${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_secret_access_key}
    camel.kamelet.aws-s3-source.region: ca-central-1

So our questions are:

1. We need to restore the topics of a single user from s3 to kafka with a specific year or month or time, so how can we do that.?
2. Once we restored the data to kafka cluster, what will happen with this connect and connector? Do we need to continue to run this on the k8s cluster?
3. This issue is also connected to my another issue: https://github.com/apache/camel-kafka-connector/issues/1347 Because the sink need to take the backup in yearly, monthly, daily and etc, so that we can restore it to kafka.

[oscerd]

The only way I see is using the prefix

camel.kamelet.aws-s3-source.prefix

You'll need to specifiy the prefix for the customer you want to restore and in the topic you'll get the content.

For question 2, I suppose you can manually stop the connector once you verified the restore has been completed.