rangatdt

Implementing an opaque transactional HdfsState

1

Thanks much for this contrib. First, could you confirm that the current HdfsState implementation is non-transactional state and so there is no guarantee that data gets written to HDFS exactly...

MoveFileRotation rotates even 0-sized files. For non-continous streams and especially when microbatching, the number of 0-sized files can become problematic. A modification to the MoveFileRotation to delete 0-sized files instead...