dagger
dagger copied to clipboard
analyze: support for multiple sinks in dagger
Context: To leave out the complexities of handling multiple sinks. We would like to integrate Dagger & Firehose in a way where all the sink logic is abstracted out in a library this will allow a consistent behavior and avoid logic duplication.
While creating the RFC, things we need to analyze for:
-
Common library abstraction for sinks
- Github name - dependencies -
Abstraction for dagger and firehose sinks.
-
Conventions
-
Common instrumentation library if possible.
-
Implications on extra serialisation from sql output to sink.
-
How to handle different schemas e.g. Proto, json, Avro.
-
Configuration list
-
Capture Any extra permissions.
-
Milestones.
-
Impact of Extra classes for sinks in dagger jars(Core, minimal etc)
-
New alerts for particular sinks.
-
Metrics required.
-
Impact on extra Kube resources required.
-
For every sink, put some thoughts on implementation.
-
check Data loss and checkpointing.