dolphinscheduler icon indicating copy to clipboard operation
dolphinscheduler copied to clipboard

[Feature-11488][Datax]Datax can submit to YARN

Open duhanmin opened this issue 3 years ago • 2 comments

https://github.com/apache/dolphinscheduler/issues/11488

Instructions:

Configure the environment variables DATAX_ON_YARN_DEPEND_JAR and DATAX_HDFS_PATH to run DATAX ON YARN

Parameter explanation: DATAX_ON_YARN_DEPEND_JAR: https://github.com/duhanmin/datax-on-yarn Compile and package

DATAX_HDFS_PATH: You need to upload the datax installation package to yarn

example:

#export DATAX_ON_YARN_DEPEND_JAR=${DATAX_ON_YARN_DEPEND_JAR:-/opt/soft/datax/datax-on-yarn-1.0.0.jar}
#DATAX_HDFS_HOME is HDFS path
#export DATAX_HDFS_PATH=${DATAX_HDFS_HOME:-/hdfs/opt/hadoop/datax.tar.gz}

duhanmin avatar Sep 22 '22 09:09 duhanmin

使用方法:

配置环境变量DATAX_ON_YARN_DEPEND_JAR和DATAX_HDFS_PATH就可以运行DATAX ON YARN

参数解释: DATAX_ON_YARN_DEPEND_JAR: https://github.com/duhanmin/datax-on-yarn 编译打包

DATAX_HDFS_PATH: 需要将datax安装包上传到yarn

例子:

#export DATAX_ON_YARN_DEPEND_JAR=${DATAX_ON_YARN_DEPEND_JAR:-/opt/soft/datax/datax-on-yarn-1.0.0.jar}
#DATAX_HDFS_HOME is HDFS path
#export DATAX_HDFS_PATH=${DATAX_HDFS_HOME:-/hdfs/opt/hadoop/datax.tar.gz}

duhanmin avatar Sep 22 '22 09:09 duhanmin

Hi @duhanmin , please associate the issue and correct the title.

SbloodyS avatar Sep 22 '22 09:09 SbloodyS

@duhanmin Hello, thanks for submitting this PR and it's a good feature. I've added some comments and please take a look.

EricGao888 avatar Sep 24 '22 16:09 EricGao888