jitsu icon indicating copy to clipboard operation
jitsu copied to clipboard

Support for alternative events queue broker

Open absorbb opened this issue 3 years ago • 0 comments

Problem of current design:

  • Events routing in Jitsu is hardcoded.
  • Also many components are hardwired. E.g. http handlers and destinations
  • Only 2 implementation of events queue supported: Redis and Inmemory. Enterprise client may what to use other solutions like Kafka

Proposals:

  • use events broker to detach component from each other
  • use abstraction to allow supporting multiple event streaming solution: kafka, redpanda, pulsar,...
  • don't reinvent routing mechanism for solution with built-in one
  • keep supporting inmemory and redis with handmade routing mechanism
  • (separate task) Expand usage of events broker for more cases: user recognition?, events cache?, sync statues, post handle destinations and notification, errors reporting, metrics reporting ( e.g. much more convenient to consolidate the most all metrics logic in one service, other service need to simply send status event )

absorbb avatar Apr 06 '22 07:04 absorbb