demo-realtime-digital-marketing icon indicating copy to clipboard operation
demo-realtime-digital-marketing copied to clipboard

Published 20 hours ago • verified publisher icon singlestore-labs

Add data ingest via kafka as an option

Open mpankow opened this issue 3 years ago • 2 comments

All of the DDL and data is there so I am hoping this would be a relatively minor enhancement. Sometimes people want kafka ingest specifically. It would be great to have that option as part of this demo.

mpankow avatar Apr 14 '22 23:04 mpankow

This is an interesting idea... the main requirement is being able to provide connection details to a kafka cluster with the correct data in the correct topics. Since the data is already available in s3 - in theory it's possible to ingest the data into kafka and then modify the pipelines accordingly.

Assuming we build something like this - which hosted kafka solution were you thinking of using? If this is primarily to unlock our sales team I can add an advanced option setting that lets you plug in kafka details - but leave it up to you guys to setup kafka correctly.

carlsverre avatar Apr 21 '22 23:04 carlsverre

The goal would be to unlock the sales team. I think leaving it to SEs to get a kafka environment setup is reasonable.

mpankow avatar Apr 22 '22 14:04 mpankow