[Bug] [upgrade] upgrade 3.2.0 ，dataX Bug

[Capricorn0010]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

运行dataX报错 [INFO] 2023-10-23 15:54:38.574 +0800 - Set taskVarPool: null successfully [INFO] 2023-10-23 15:54:38.574 +0800 - *********************************************************************************************** [INFO] 2023-10-23 15:54:38.574 +0800 - ********************************* Execute task instance ************************************* [INFO] 2023-10-23 15:54:38.574 +0800 - *********************************************************************************************** [INFO] 2023-10-23 15:54:38.575 +0800 - Final Shell file is : #!/bin/bash BASEDIR=$(cd dirname $0; pwd) cd $BASEDIR ${PYTHON_LAUNCHER} ${DATAX_LAUNCHER} --jvm="-Xms1G -Xmx1G" -p "-Dschedule_timezone='Asia/Shanghai' -Dsystem.task.definition.name='sp1_ods_mes_bianYuan_data_hi' -Dsystem.project.name='null' -Dsystem.project.code='9263644055840' -Dsystem.workflow.instance.id='21262' -Dsystem.biz.curdate='20231023' -Dsystem.biz.date='20231022' -Dsystem.task.instance.id='326795' -Dsystem.workflow.definition.name='mes_gas_exhaustion' -Dsystem.task.definition.code='9672418489122' -Dsystem.workflow.definition.code='9672418489125' -Dsystem.datetime='20231023155438'" /tmp/dolphinscheduler/exec/process/default/9263644055840/9672418489125_7/21262/326795/21262_326795_job.json [INFO] 2023-10-23 15:54:38.576 +0800 - Executing shell command : sudo -u dolphinscheduler -i /tmp/dolphinscheduler/exec/process/default/9263644055840/9672418489125_7/21262/326795/21262_326795.sh [INFO] 2023-10-23 15:54:38.579 +0800 - process start, process id is: 70912 [INFO] 2023-10-23 15:54:39.579 +0800 - -> /tmp/dolphinscheduler/exec/process/default/9263644055840/9672418489125_7/21262/326795/21262_326795.sh: line 4: --jvm=-Xms1G -Xmx1G: command not found

What you expected to happen

3.2.0版本运行dataX组件，使用了两个变量？${PYTHON_LAUNCHER} ${DATAX_LAUNCHER}

PYTHON_LAUNCHER和 3.1.X 版本的PYTHON_HOME 有区别吗

How to reproduce

upgrade 3.2.0

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[github-actions[bot]]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

运行dataX报错 [INFO] 2023-10-23 15:54:38.574 +0800 - Set taskVarPool: null successfully [INFO] 2023-10-23 15:54:38.574 +0800 - *********************************************************************************************** [INFO] 2023-10-23 15:54:38.574 +0800 - ********************************* Execute task instance ************************************* [INFO] 2023-10-23 15:54:38.574 +0800 - *********************************************************************************************** [INFO] 2023-10-23 15:54:38.575 +0800 - Final Shell file is : #!/bin/bash BASEDIR=$(cd dirname $0; pwd) cd $BASEDIR ${PYTHON_LAUNCHER} ${DATAX_LAUNCHER} --jvm="-Xms1G -Xmx1G" -p "-Dschedule_timezone='Asia/Shanghai' -Dsystem.task.definition.name='sp1_ods_mes_bianYuan_data_hi' -Dsystem.project.name='null' -Dsystem.project.code='9263644055840' -Dsystem.workflow.instance.id='21262' -Dsystem.biz.curdate='20231023' -Dsystem.biz.date='20231022' -Dsystem.task.instance.id='326795' -Dsystem.workflow.definition.name='mes_gas_exhaustion' -Dsystem.task.definition.code='9672418489122' -Dsystem.workflow.definition.code='9672418489125' -Dsystem.datetime='20231023155438'" /tmp/dolphinscheduler/exec/process/default/9263644055840/9672418489125_7/21262/326795/21262_326795_job.json [INFO] 2023-10-23 15:54:38.576 +0800 - Executing shell command : sudo -u dolphinscheduler -i /tmp/dolphinscheduler/exec/process/default/9263644055840/9672418489125_7/21262/326795/21262_326795.sh [INFO] 2023-10-23 15:54:38.579 +0800 - process start, process id is: 70912 [INFO] 2023-10-23 15:54:39.579 +0800 - -> /tmp/dolphinscheduler/exec/process/default/9263644055840/9672418489125_7/21262/326795/21262_326795.sh: line 4: --jvm=-Xms1G -Xmx1G: command not found

What you expected to happen

3.2.0版本运行dataX组件，使用了两个变量？${PYTHON_LAUNCHER} ${DATAX_LAUNCHER}

PYTHON_LAUNCHER和 3.1.X 版本的PYTHON_HOME 有区别吗

How to reproduce

upgrade 3.2.0

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[icefery]

I have also encountered the same problem. May I ask if there is any solution?

[dufeng1010]

export DATAX_LAUNCHER and PYTHON_LAUNCHER in /etc/profile in linux. after that, execute datax task, it works. but i think it should 'source dolphinscheduler_env.sh' in datax script file. export DATAX_LAUNCHER and PYTHON_LAUNCHER in dolphinscheduler_env.sh。

[ztx-go]

export DATAX_LAUNCHER and PYTHON_LAUNCHER in /etc/profile in linux. after that, execute datax task, it works. but i think it should 'source dolphinscheduler_env.sh' in datax script file. export DATAX_LAUNCHER and PYTHON_LAUNCHER in dolphinscheduler_env.sh。

I also encountered this problem, and finally implemented datax successfully according to your solution，very good

[heichong]

I encountered the same problem, is this a bug？

[guodachao]

export DATAX_LAUNCHER and PYTHON_LAUNCHER in /etc/profile in linux. after that, execute datax task, it works. but i think it should 'source dolphinscheduler_env.sh' in datax script file. export DATAX_LAUNCHER and PYTHON_LAUNCHER in dolphinscheduler_env.sh。

亲测有效，折腾死我了

[icefery]

export DATAX_LAUNCHER and PYTHON_LAUNCHER in /etc/profile in linux. after that, execute datax task, it works. but i think it should 'source dolphinscheduler_env.sh' in datax script file. export DATAX_LAUNCHER and PYTHON_LAUNCHER in dolphinscheduler_env.sh。

According to the log, datax task uses sudo -u root -i <SH_FILE> to execute, and <SH_FILE> can not get default envinonment variables.

I found creating an Environment and using it while creating Datax Task could also work well, this might be more helpful in container environment than /etc/profile.

export JAVA_HOME=/opt/java/openjdk
export PYTHON_LAUNCHER=/opt/soft/python/bin/python
export DATAX_LAUNCHER=/opt/soft/datax/bin/datax.py
export PATH=$JAVA_HOME/bin:$PATH

https://github.com/apache/dolphinscheduler/blob/e648d6d2adede44c11711c313b5f27d4474961c5/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java#L137

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.