[Feature ]Support multiple Hadoop clusters

[binbinCheng]

描述：支持多Hadoop集群,运维优化、降本 需求内容

1. 支持多hadoop集群
2. 结果集支持多集群读取
3. linkisManager支持跨集群选择ECM、EC
4. 支持跨集群进行计算

---

Description: Support multiple Hadoop clusters, optimize operation and maintenance, and reduce costs Required content

1. Support multiple hadoop clusters
2. The result set supports multi-cluster reading
3. linkisManager supports cross-cluster selection of ECM and EC
4. Support cross-cluster computing

[github-actions[bot]]

:blush: Welcome to the Apache Linkis community!!

We are glad that you are contributing by opening this issue.

Please make sure to include all the relevant context. We will be here shortly.

If you are interested in contributing to our website project, please let us know! You can check out our contributing guide on :point_right: How to Participate in Project Contribution.

Community

WeChat Assistant	WeChat Public Account

Mailing Lists

Name	Description	Subscribe	Unsubscribe	Archive
[email protected]	community activity information	subscribe	unsubscribe	archive

[chenghuichen]

Hi @peacewong , we are going to sovle this issue, could you assign this to me? Thanks a lot.

[peacewong]

Hi @peacewong , we are going to sovle this issue, could you assign this to me? Thanks a lot.

alright, thank you very much

[chenghuichen]

Storage design in multi-cluster mode:

• introduce an explicit concept linkis hadoop-conf of linkis default-cluster, to distinguish the job hadoop-conf of job Cluster. The conf dir may be the same when in single-cluster mode and both read from system env; but in multi-cluster mode, an explicit config of linkis hadoop-conf dir must be set, representing the chosen linkis default-cluster;
• module linkis-storage will purely use linkis hadoop-conf to init its filesystem. So in EC and ECM, the resultSet, JarUDF and BmlResource will read and write from storage of linkis default-cluster, rather than storage of its job cluster.