bitsail
bitsail copied to clipboard
[Improve][HDFS] Support HDFS incremental read
Description
HDFS Inputformat supports passing in path parameters and imports all files under the path. However, in incremental scenarios, it is necessary to filter based on file name or file creation time
In the implementation, you can customize the PathFilter strategy, filter according to the file name regular match or file creation time, and set it through the setInputPathFilter function
BitSail Component or Code Module
BitSail Connector
Are you willing to submit PR?
- [ ] Yes, I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
@lichang-bd I want to have a try. assign me please.