conduit
conduit copied to clipboard
ConduitIO
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
The idea is to simplify the [`Destination` interface](https://github.com/ConduitIO/conduit-connector-sdk/blob/733149cddc0bcdfd1144012f1bf41651355653d1/destination.go#L38) in the SDK and move complexity related to asynchronous writes to the SDK. We need to write a proper design document around...
### Feature description Currently, it's not possible for JS transforms to return records with changed structured data. This issue has two parts: 1. There's no way in a JS script...
Change all existing connectors to use the newest version of the SDK and use the new record. Built-in: - [x] [File](https://github.com/ConduitIO/conduit-connector-file) https://github.com/ConduitIO/conduit-connector-file/pull/14 - [x] [Kafka](https://github.com/ConduitIO/conduit-connector-kafka) https://github.com/ConduitIO/conduit-connector-kafka/pull/38 - [x] [Postgres](https://github.com/ConduitIO/conduit-connector-postgres) https://github.com/ConduitIO/conduit-connector-postgres/pull/49...
While everyone should do their own benchmarking, we need to give developers some where to start. This will anchor expectations and then we can help folks tune their installations as...
Subtasks: - [x] #507 - [x] #508 - [x] #509 - [x] #510 - [ ] #512 - [ ] #511 This is going to be a breaking change. Can...
We need to implement logic that takes a pipeline config file and provisions the pipeline defined in it. If a pipeline with that name already exists it needs to be...
### Description This PR optimizes the semaphore. The semaphore is on the hot path since every message needs to acquire it, an optimization will directly affect the performance of the...
### Feature description Currently, we're using bash scripts to configure a Docker container (in which Conduit will be running) and also to create test pipelines. There are two implications of...
### Feature description The usage of built-in vs standalone plugins affects performance test results (built-in ones affect Conduit's resource usage, standalone plugins incur the cost of gRPC usage). We should...
### Bug description This was discovered while working on the performance benchmarks. Records larger than 4 MB in size cannot be streamed through a pipeline, if they've been read with...