bitsail icon indicating copy to clipboard operation
bitsail copied to clipboard

[Improve][HDFS] Support HDFS incremental read

Open lichang-bd opened this issue 2 years ago • 1 comments

Description

HDFS Inputformat supports passing in path parameters and imports all files under the path. However, in incremental scenarios, it is necessary to filter based on file name or file creation time

image

In the implementation, you can customize the PathFilter strategy, filter according to the file name regular match or file creation time, and set it through the setInputPathFilter function

image image

BitSail Component or Code Module

BitSail Connector

Are you willing to submit PR?

  • [ ] Yes, I am willing to submit a PR!

Code of Conduct

lichang-bd avatar Feb 13 '23 13:02 lichang-bd

@lichang-bd I want to have a try. assign me please.

xianrenzw avatar Oct 20 '24 06:10 xianrenzw