dagger icon indicating copy to clipboard operation
dagger copied to clipboard

analyze: support for multiple sinks in dagger

Open gauravsinghania opened this issue 3 years ago • 0 comments

Context: To leave out the complexities of handling multiple sinks. We would like to integrate Dagger & Firehose in a way where all the sink logic is abstracted out in a library this will allow a consistent behavior and avoid logic duplication.

While creating the RFC, things we need to analyze for:

  • Common library abstraction for sinks

    -  Github name
    - dependencies 
    
  • Abstraction for dagger and firehose sinks.

  • Conventions

  • Common instrumentation library if possible.

  • Implications on extra serialisation from sql output to sink.

  • How to handle different schemas e.g. Proto, json, Avro.

  • Configuration list

  • Capture Any extra permissions.

  • Milestones.

  • Impact of Extra classes for sinks in dagger jars(Core, minimal etc)

  • New alerts for particular sinks.

  • Metrics required.

  • Impact on extra Kube resources required.

  • For every sink, put some thoughts on implementation.

  • check Data loss and checkpointing.

gauravsinghania avatar Mar 28 '22 13:03 gauravsinghania