Functionality Questions about how customized data extraction can get

[switzer]

Hi - I have some functionality questions about Bifrost:

1. Can Bifrost output data onto S3 in a format that is consumable by Redshift?
2. How often can files be written out? Every hour? Every minute?
3. Can Bifrost extract data by a data field in the Kafka data, rather than the Kafka created_at timestamp?

Thanks!

[kleinron]

as for [2] - it's configurable

[tgk]

Hi @switzer - sorry about the late reply!

1. It cannot. There is no manipulation performed and the output format is .baldr. We have been thinking about adding support for other file types. In your case, a new-line delimited file would be a solution. It would be up to the data generators that publish things to Bifrost to escape newline characters in that case.
2. As @kleinron says, it's configurable
3. Not at the moment. Bifrost is message-type agnostic, which means it works with JSON, comma-delimited messages, Protocol buffers - anything you throw at it. We are not keen on leaking message interpretation out. A plugin system could help you out but we have no immediate plans of implementing one.