alpakka
alpakka copied to clipboard
Generating time-based path for each new file
Short description
It's about a possibility to create a file path for each new file depending on timestamp supplied in hdfs messages.
Details
Currently the only possibility is to have file paths created from current time in milliseconds and a row count. Moreover, each file will be written into the same folder defined at flow-creation time. So, it would be great to have a possibility to create a file path (including its folder name) for each new file depending on timestamp supplied in hdfs messages. It's the real business case I've stumble upon my current project.
PS. I already have an implementation for this and could proceed with creating PR, just wanted to know what other people thoughts could be on the matter.
Sounds reasonable. We would welcome a PR. It might make sense to make the destination path more generic. For example, providing a user defined function to generate a path, given the HDFS message/metadata.