snowplow-rdb-loader icon indicating copy to clipboard operation
snowplow-rdb-loader copied to clipboard

Published 20 hours ago • verified publisher icon snowplow

Transformer-kafka to support AWS S3

Open sivankumar86 opened this issue 1 year ago • 0 comments

It would be great if transformer kafka can read from kafka and write to s3.

We deployed snowplow on EKS but , we want to use kafka instead of kinesis hence, Could you add support to read from enrich and write to s3 then load to snowflake.

Enrichment (KAFKA) ==> Transformer (s3) ==> kafka ==> Loader ==> s3

Transformer load to s3 then post the load event to kafka then loader listen kafka then load to snowflake from s3 in batch mode.

sivankumar86 avatar Jun 24 '24 06:06 sivankumar86