bitsail
bitsail copied to clipboard
bytedance
[Feature][Connector][File-Connector] Support multi file systems
Now bitsail not support some file systems like oss s3 hdfs localfile sftp ftp and so on. I want to do this feature.
Welcome to cooperate and build together. The community will also send gifts to contributors, and discuss any questions at any time @TyrantLucifer
@TyrantLucifer Hi, thanks for raising this issue. Which connector you are referring to? Do you mean the hive or the streaming file system sink?
@TyrantLucifer Hi, thanks for raising this issue. Which connector you are referring to? Do you mean the hive or the streaming file system sink?
Hi @garyli1019, this means that I want do develop all file connectors and refactor it. As we now, every filesystem has their own access method and different authentication method. If we want to implement multi file connectors we need implement different access logic. In order to solve this problem and unified code stucture, using hdfs interface is a better choice. What do you think about it?
Sorry for the delay because of the confirmed COVID-19, I will continue this task this week.
@TyrantLucifer Do you mean something like an abstraction FileSystem layer to handle the different logic? That's definitely very useful. Currently the hive legacy sink connector is using Flink FileSystem and the streaming file sink is using Hadoop FileSystem. We are planning to unified this as well. This project has a big impact and will touch many modules, so I think we can discuss further before we start the implementation. But before that, please take a good rest! cc @hk-lrzy @love-star
@TyrantLucifer Do you mean we can refactor all FileSystem use one interface like link. I'm also interest in it, if there is anything I can help, just discuss with me. I have made a try, but in my try, I find it that in our hadoop legacy module, we use the org.apache.hadoop.mapred.InputFormat to read the hdfs file, when I want to combine it with other filesystem, I don't known how to adapt this interface to other filesystem. If you encounter any question in your try, or you want someone help you so some little thing, I can help you.
@TyrantLucifer Are you still working on this task? if there is any problem you meet, just leave comments and I will response as soon as possible.
@TyrantLucifer Are you still working on this task? if there is any problem you meet, just leave comments and I will response as soon as possible.
Yup, please give me some time. Thanks!