dolphinscheduler icon indicating copy to clipboard operation
dolphinscheduler copied to clipboard

[Bug] When [Module Name] contains multiple sub-processes, the task instance in the sub-process instance obtains duplicate parameters of the pre-task instance.

Open AlanZQ opened this issue 1 year ago • 13 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

在工作流中,创建多个子流程,例如创建12个子流程,子流程在获取上游任务参数时,会出现覆盖/错误/以及12个节点拿到的参数完全一致,这不符合预期 通过数据库中查询来看,生成的12的子流程实例,参数没有问题,但是在varpool的设置阶段,获取parent_process_instance_id所对应的varpool,是相同的,这导致子流程实例中的任务实例,执行时的参数永远是相同的

iShot_2024-12-05_10 44 15

What you expected to happen

通过数据库中查询来看,生成的12的子流程实例,参数没有问题,但是在varpool的设置阶段,获取parent_process_instance_id所对应的varpool,是相同的,这导致子流程实例中的任务实例,执行时的参数永远是相同的 通过debug setSubProcessParam这个方法,能够看出每个子流程任务实例获取到的varpool参数都有问题,并且会随机出现重复

How to reproduce

我已经验证了3.2.1的standalone镜像,3.2.2的standalone镜像,debug镜像代码,都存在此问题,也可以在这两个镜像web服务页面,复现。我提供了我的验证工作流json,⚠️注意:导入后需要更改主流程中子流程的引用为新的id workflow_1733366901095.json

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

AlanZQ avatar Dec 05 '24 02:12 AlanZQ

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

In the workflow, create multiple sub-processes, for example, create 12 sub-processes. When the sub-process obtains the parameters of the upstream task, there will be coverage/error/ and the parameters obtained by the 12 nodes are exactly the same, which is not as expected. Judging from the query in the database, there is no problem with the parameters of the generated 12 sub-process instances. However, in the varpool setting phase, the varpool corresponding to the parent_process_instance_id is obtained, which results in the task instance in the sub-process instance being executed. Parameters are always the same

iShot_2024-12-05_10 44 15

What you expected to happen

Judging from the query in the database, there is no problem with the parameters of the generated 12 sub-process instances. However, in the varpool setting phase, the varpool corresponding to the parent_process_instance_id is obtained, which results in the task instance in the sub-process instance being executed. Parameters are always the same Through the debug setSubProcessParam method, it can be seen that there is a problem with the varpool parameters obtained by each sub-process task instance, and duplicates occur randomly.

How to reproduce

I have verified that this problem exists in the standalone image of 3.2.1, the standalone image of 3.2.2, and the debug image code. It can also be reproduced on the two image web service pages. I provided my verification workflow json, ⚠️Note: After importing, you need to change the reference of the sub-process in the main process to the new id workflow_1733366901095.json

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

github-actions[bot] avatar Dec 05 '24 02:12 github-actions[bot]

Please using english to describe.

SbloodyS avatar Dec 05 '24 13:12 SbloodyS

Please using english to describe.

In the workflow, create multiple sub-processes, for example, create 12 sub-processes. When the sub-process obtains the parameters of the upstream task, there will be coverage/error/ and the parameters obtained by the 12 nodes are exactly the same, which is not as expected. Judging from the query in the database, there is no problem with the parameters of the generated 12 sub-process instances. However, in the varpool setting phase, the varpool corresponding to the parent_process_instance_id is obtained, which results in the task instance in the sub-process instance being executed. Parameters are always the same

iShot_2024-12-05_10 44 15

I have verified that this problem exists in the standalone image of 3.2.1, the standalone image of 3.2.2, and the debug image code. It can also be reproduced on the two image web service pages. I provided my verification workflow json, ⚠️Note: After importing, you need to change the reference of the sub-process in the main process to the new id

AlanZQ avatar Dec 06 '24 01:12 AlanZQ

Can you try dev to see if this issue exists?

SbloodyS avatar Dec 06 '24 02:12 SbloodyS

Can you try dev to see if this issue exists?

sure, off course

AlanZQ avatar Dec 06 '24 02:12 AlanZQ

Can you try dev to see if this issue exists?

ok, I will try later

AlanZQ avatar Dec 06 '24 02:12 AlanZQ

Can you try dev to see if this issue exists?

Encountered an error logging into the dev image.

It is already the latest version.

Index digest sha256:6b9c76ee279fb25be83521af5d7b60c19a285672f6963b52eb483698fb90d55b

image

AlanZQ avatar Dec 09 '24 06:12 AlanZQ

Is this issue fixed? I will reopen this issue.

ruanwenjun avatar Dec 11 '24 11:12 ruanwenjun

Is this issue fixed? I will reopen this issue. No, I have verified that both version 3.2.2 and the latest version have this issue. The dev version cannot log in correctly and will throw an error.

AlanZQ avatar Dec 13 '24 08:12 AlanZQ

Is this issue fixed? I will reopen this issue. No, I have verified that both version 3.2.2 and the latest version have this issue. The dev version cannot log in correctly and will throw an error.

I reopen this issue. BTW, this is a known problem, when this feature designed, I have pointed this problem, right now all varpool are stored at the workflow instance, this means once multiple varpools has the same key, then no one can know the value, and only one value will be overwrite.

ruanwenjun avatar Dec 13 '24 14:12 ruanwenjun

Is this issue fixed? I will reopen this issue. No, I have verified that both version 3.2.2 and the latest version have this issue. The dev version cannot log in correctly and will throw an error.

I reopen this issue. BTW, this is a known problem, when this feature designed, I have pointed this problem, right now all varpool are stored at the workflow instance, this means once multiple varpools has the same key, then no one can know the value, and only one value will be overwrite.

Will this be included in the community's future development plans? Subprocesses are a very important feature, and template-based workflow design heavily relies on this.

AlanZQ avatar Dec 17 '24 06:12 AlanZQ

Is this issue fixed? I will reopen this issue. No, I have verified that both version 3.2.2 and the latest version have this issue. The dev version cannot log in correctly and will throw an error.

I reopen this issue. BTW, this is a known problem, when this feature designed, I have pointed this problem, right now all varpool are stored at the workflow instance, this means once multiple varpools has the same key, then no one can know the value, and only one value will be overwrite.

Will this be included in the community's future development plans? Subprocesses are a very important feature, and template-based workflow design heavily relies on this.

I also hope this feature can be refactored, the varpool generated should contain the workflowName and taskName. If we want to use a parameterA generate by workflowA taskA, then we should can ${workflowA.taskA.parameterA}, then the parameter will be unique.

ruanwenjun avatar Dec 19 '24 01:12 ruanwenjun

Is this issue fixed? I will reopen this issue. No, I have verified that both version 3.2.2 and the latest version have this issue. The dev version cannot log in correctly and will throw an error.

I reopen this issue. BTW, this is a known problem, when this feature designed, I have pointed this problem, right now all varpool are stored at the workflow instance, this means once multiple varpools has the same key, then no one can know the value, and only one value will be overwrite.

Will this be included in the community's future development plans? Subprocesses are a very important feature, and template-based workflow design heavily relies on this.

I also hope this feature can be refactored, the varpool generated should contain the workflowName and taskName. If we want to use a parameterA generate by workflowA taskA, then we should can ${workflowA.taskA.parameterA}, then the parameter will be unique.

Got it, and thank you for your explanation. Based on my current experience, this issue does not occur when using a single sub-process. In my scenario, I modified the implementation of multiple sub-processes to use a single sub-process, which already meets the requirements of my use case. For this functionality, I hope there will be a better solution in the future. image

AlanZQ avatar Dec 19 '24 08:12 AlanZQ