Possibility to have subflow fully included in flow (template-like behaviour)

[Ben8t]

Feature description

Problem

With the deprecation of templates, there are some flaws arising with the only use of Subflow:

1. It doubles the number of execution. With templates, tasks were passed directly in the flow, like if they were written directly inside it. Before 1 flow = 1 exec. Now with subflow 1 flow = 2 exec. Making the executions lists overwhelming in the UI.

2. Some users track billing information, with 1 namespace = 1 project. Using subflow here kind of make this billing wrong as one subflow can be used in many namespaces (so there is an overlap sometimes in the executions).

3. Subflow have to be in the same namespace to have access to the same right (service account)

Possible Solutions

• For the execution problem (1. & 2.):
  • can be handle with labels or filtering: bring a flag subflow on the flow, that allow to hide subflow from the list
  • add a included:true property to the io.kestra.core.tasks.flows.Flow :

id: "flow"
type: "io.kestra.core.tasks.flows.Flow"
namespace: dev
flowId: subflow
inputs:
  user: "Rick Astley"
  favorite_song: "Never Gonna Give You Up"
wait: true
transmitFailed: true
included: true 

Without this kind of flag, currently the flow will create two executions: one for the main flow, one for the subflow. The included property let the user to choose if the subflow can be "included" directly in the subflow - merging the old template behaviour into the subflow one; i.e. having only one execution for the thing.

• For the access control (3.):
  • should be handle in this epic https://github.com/kestra-io/kestra-ee/issues/427

[anna-geller]

Hiding subflows from the UI (a dedicated filter property showing whether some execution is a subflow execution or not) is something we can definitely add: https://github.com/kestra-io/kestra/issues/2559

Regarding a secret/service account as input property, you're spot on, we will do it as part of that linked issue.

I worry about not creating a proper child execution for a subflow. E.g. how would it look like when such a subflow would be used on the ForEachItem? It may lead to many issues. Let's handle these two at first.

[npranav10]

Hi Kestra Team,

I was going to open a similar issue. @Ben8t has perfectly described my thoughts about point 1. Any updates on this feature request ?

[anna-geller]

@npranav10 you can see the estimate when something will be released in the Milestone on the right side below labels, this one is planned for 0.18 atm.