stream-reactor
stream-reactor copied to clipboard
lensesio
A collection of open source Apache 2.0 Kafka Connector maintained by Lenses.io.
Join us on slack 
Lenses for Apache Kafka
Lenses offers SQL (for data browsing and Kafka Streams), Kafka Connect connector management, cluster monitoring and more.
You can find more on lenses.io
Stream Reactor
A collection of components to build a real time ingestion pipeline.
Kafka Compatibility
From version 4.0.0 separate artifacts are built targeting Kafka 3.1 and Kafka 2.8.
- Kafka 3.1 (Confluent 7.1) - Stream Reactor 4.0.0 (Kafka 3.1 Build)
- Kafka 2.8 (Confluent 6.2) - Stream Reactor 4.0.0 (Kafka 2.8 Build)
- Kafka 2.5 (Confluent 5.5) - Stream reactor 2.0.0+
- Kafka 2.0 -> 2.4 (Confluent 5.4) - Stream reactor 1.2.7
DEPRECATION NOTICE
The following connectors have been deprecated and are no longer included in any release from 3.0.
- Blockchain
- Bloomberg
- Rethink
- VoltDB
The following connectors have been deprecated and are no longer included in any release from 4.0.
- Coap
- Hive 1.1
Connectors
Please take a moment and read the documentation and make sure the software prerequisites are met!!
| Connector | Type | Description | Docs |
|---|---|---|---|
| AWS S3 | Sink | Copy data from Kafka to AWS S3. | Docs |
| AzureDocumentDb | Sink | Copy data from Kafka and Azure Document Db. | Docs |
| Cassandra | Source | Copy data from Cassandra to Kafka. | Docs |
| *Cassandra | Sink | Certified DSE Cassandra, copy data from Kafka to Cassandra. | Docs |
| Elastic 6 | Sink | Copy data from Kafka to Elastic Search 6.x w. tcp or http | Docs |
| FTP/HTTP | Source | Copy data from FTP/HTTP to Kafka. | Docs |
| Hazelcast | Sink | Copy data from Kafka to Hazelcast. | Docs |
| HBase | Sink | Copy data from Kafka to HBase. | Docs |
| Hive | Source | Copy data from Hive/HDFS to Kafka. | Docs |
| Hive | Sink | Copy data from Kafka to Hive/HDFS | Docs |
| InfluxDb | Sink | Copy data from Kafka to InfluxDb. | Docs |
| Kudu | Sink | Copy data from Kafka to Kudu. | Docs |
| JMS | Source | Copy data from JMS topics/queues to Kafka. | Docs |
| JMS | Sink | Copy data from Kafka to JMS. | Docs |
| MongoDB | Sink | Copy data from Kafka to MongoDB. | Docs |
| MQTT | Source | Copy data from MQTT to Kafka. | Docs |
| MQTT | Sink | Copy data from Kafka to MQTT. | Docs |
| Pulsar | Source | Copy data from Pulsar to Kafka. | Docs |
| Pulsar | Sink | Copy data from Kafka to Pulsar. | Docs |
| Redis | Sink | Copy data from Kafka to Redis. | Docs |
Release Notes
Please see the Stream Reactor Release Notes at Lenses Documentation.
Building
To build:
sbt clean compile
To test:
sbt test
To create assemblies:
sbt assembly
To build a particular project for kafka 2.8:
sbt "project cassandra-kafka-2-8" compile
To build a particular project for kafka 3.1:
sbt "project cassandra-kafka-3-1" compile
To test a particular project:
sbt "project cassandra-kafka-2-8" test
To create a jar of a particular project:
sbt "project cassandra-kafka-2-8" assembly
Running E2E tests
If not already built, you must first build the connector archives:
sbt "project cassandra-kafka-2-8" assembly
sbt "project elastic6-kafka-2-8" assembly
sbt "project mongodb-kafka-2-8" assembly
sbt "project redis-kafka-2-8" assembly
sbt "project cassandra-kafka-3-1" assembly
sbt "project elastic6-kafka-3-1" assembly
sbt "project mongodb-kafka-3-1" assembly
sbt "project redis-kafka-3-1" assembly
To run the tests:
sbt e2e:test
Contributing
We'd love to accept your contributions! Please use GitHub pull requests: fork the repo, develop and test your code, semantically commit and submit a pull request. Thanks!
License
lensesio