[flyte-bug] Can't retrieve the nodes of a dynamic workflow in an execution by FlyteRemote

[kuni23]

Describe the bug

Hi all! I am trying to reach the nodes of a dynamic workflow in an execution by FlyteRemote, but it is not working.

Here are the steps I've taken:

Fetch the execution using FlyteRemote: exec = remote.fetch_execution(project, domain, execution_name)

Retrieve the ID of the dynamic node execution: dynamic_node_exec_id = remote.client.list_node_executions(exec.id)[0][3].id

Obtain the ID of the dynamic task execution: dynamic_task_id = remote.client.list_task_executions_paginated(dynamic_node_exec_id)[0][0].id

Attempt to list node executions for the task, but I get an empty list: spawned_nodes = remote.client.list_node_executions_for_task_paginated(dynamic_task_id)

Expected behavior

Since I have got the task execution of a dynamic workflow, applying the list_node_executions_for_task_paginated should return its node executions in a list according to the documentation of FlyteRemote.

Additional context to reproduce

No response

Screenshots

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[welcome[bot]]

Thank you for opening your first issue here! 🛠

[dosubot[bot]]

🤖

Based on the information you've provided, it seems like you're trying to retrieve all nodes of a dynamic workflow using the list_node_executions_for_task_paginated method. However, this method only returns the nodes that were launched by a specific task execution. This behavior is demonstrated in the test TestCreateChildNodeExecutionForTaskExecution in the flyteadmin/tests/node_execution_test.go file.

If you're trying to retrieve all nodes of a dynamic workflow, you might want to use the ListNodeExecutions method instead, which lists all node executions for a given workflow execution. Please note that this method only returns top-level nodes for a workflow, not the child nodes launched by a specific task execution.

Regarding the list_node_executions_for_task_paginated method in the FlyteRemote class, it's not directly mentioned in the provided context. However, based on the context, it likely relates to the ListNodeExecutionsForTask method in the AdminService class. This method fetches a list of NodeExecution instances launched by the referenced TaskExecution. Currently, it raises a NotImplementedError exception, indicating that the method is not yet implemented. You can find more details in the flyteidl/gen/pb_python/flyteidl/service/admin_pb2_grpc.py file.

I hope this helps clarify the issue. If you have further questions or need more clarification, please let me know.

Sources

• flyteadmin/tests/node_execution_test.go
• flyteidl/gen/pb_python/flyteidl/service/admin_pb2_grpc.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[kuni23]

@dosu-bot

I believe ListNodeExecutionsForTask is implemented. You are right that it raises NotImplementedError exception here since it is only the interface of the method: https://github.com/flyteorg/flyte/blob/master/flyteidl/gen/pb_python/flyteidl/service/admin_pb2_grpc.py#L496 But the method is implemented here: https://github.com/flyteorg/flyte/blob/master/flyteidl/gen/pb_python/flyteidl/service/admin_pb2_grpc.py#L1421

I would like to have ListNodeExecutionsForTask to work since as you said ListNodeExecutions method is not working for dynamic workflows. And I believe this functionality works well on the flyte dashboard since I can see the nodes of a dynamic workflow and their outputs there. But by FlyteRemote in Python I cannot retrieve them.

Could a human maintainer review my issue as well, who might have more insight into this matter?

[dosubot[bot]]

🤖

Thank you for providing the detailed steps and context. It seems like you're encountering an issue with the ListNodeExecutionsForTask method.

From the context provided, it appears that the methods GetNodeExecution, GetDynamicNodeWorkflow, ListNodeExecutions, and ListNodeExecutionsForTask are all raising NotImplementedError exceptions. This means that these methods are declared but not yet provided with the functional implementation. Here is the relevant code:

def GetNodeExecution(self, request, context):
    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
    context.set_details('Method not implemented!')
    raise NotImplementedError('Method not implemented!')

def GetDynamicNodeWorkflow(self, request, context):
    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
    context.set_details('Method not implemented!')
    raise NotImplementedError('Method not implemented!')

def ListNodeExecutions(self, request, context):
    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
    context.set_details('Method not implemented!')
    raise NotImplementedError('Method not implemented!')

def ListNodeExecutionsForTask(self, request, context):
    context.set_code(grpc.StatusCode.UNIMPLEMENTED)
    context.set_details('Method not implemented!')
    raise NotImplementedError('Method not implemented!')

Each of these methods is currently set to raise a NotImplementedError with the message 'Method not implemented!'. This is why you're unable to retrieve the nodes of a dynamic workflow in an execution by FlyteRemote. You'll need to wait for these methods to be implemented or implement them yourself if you have the necessary permissions and knowledge.

As for the ListNodeExecutionsForTask method, it is not present in the provided context, so I cannot provide the exact functionality of this method.

I hope this clarifies the issue you're facing. If you have any more questions or need further clarification, feel free to ask.

Sources

• flyteidl/gen/pb_python/flyteidl/service/admin_pb2_grpc.py
• flyteadmin/pkg/manager/impl/node_execution_manager.go

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[arbaobao]

#take

[yubofredwang]

Hi @arbaobao any update on this one?

[github-actions[bot]]

"Hello 👋, this issue has been inactive for over 90 days. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏"

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 90 days and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

[btang-stripe]

Re-opening as we'd like to see this feature too. A lot of our users have dynamically generated nodes in their workflows and we currently only get the top level node returned when querying node_executions

[github-actions[bot]]

"Hello 👋, this issue has been inactive for over 90 days. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏"

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 90 days and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏