alpakka icon indicating copy to clipboard operation
alpakka copied to clipboard

Generating time-based path for each new file

Open bondarenko opened this issue 3 years ago • 1 comments

Short description

It's about a possibility to create a file path for each new file depending on timestamp supplied in hdfs messages.

Details

Currently the only possibility is to have file paths created from current time in milliseconds and a row count. Moreover, each file will be written into the same folder defined at flow-creation time. So, it would be great to have a possibility to create a file path (including its folder name) for each new file depending on timestamp supplied in hdfs messages. It's the real business case I've stumble upon my current project.

PS. I already have an implementation for this and could proceed with creating PR, just wanted to know what other people thoughts could be on the matter.

bondarenko avatar Feb 21 '21 22:02 bondarenko

Sounds reasonable. We would welcome a PR. It might make sense to make the destination path more generic. For example, providing a user defined function to generate a path, given the HDFS message/metadata.

seglo avatar Feb 22 '21 15:02 seglo