dagger
dagger copied to clipboard
raystack
Dagger is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of real-time streaming data.
Context: To leave out the complexities of handling multiple sinks. We would like to integrate Dagger & [Firehose](https://github.com/odpf/firehose) in a way where all the sink logic is abstracted out in...
As part of this issue, want to add support for handling multiple streams for Parquet Data Source. That is, users should be able to specify multiple GCS URLs. Dagger should...
Extra fields present in the parquet data not present in the protobuf schema will be ignored. However, it might be possible that: - there are some fields in protobuf schema...
As part of this issue, we want to add support for configuring a split assigner which can assign splits to source readers in an `almost-deterministic` order as decided by the...
**Acceptance Criteria** - Add Basic BigQuery Sink using the sink-connector APIs. - Configure Proto/JSON Serialiser to convert Row into OdpfMessage - Call sink APIs to push messages to BQ. **Out...
**Context:** As a first task, we want to create a repo which will hold the sink-connector library. This library will be used by both firehose and dagger. it might be...
Currently, there are no null checks for required params coming from the run job API payload. Add support to check and handle them with proper error messages.
Currently, the HashTransformer uses its own implementation for protobuf SerDe. * Move SerDe logic from dagger-core to dagger-common. * Use common SerDe for both core and HashTransformer.