[FEATURE]: Option for re-resolving configuration between workflow steps

[carolinapc]

Bug

Current Behavior

This calculatePathToRoot script bellow creates a garden.env file adding pathToRoot env variable and it is called by the first step in the workflow bellow. The variable is used inside the template module linked to the service that is being deployed.

When we run the workflow for the first time, the garden.env file is created but garden can not find the pathToRoot variable, throwing the error: Invalid template string (...): Could not find key pathToRoot under var...

The the second time we run it works, because we already have the garden.env created.

calculatePathToRoot.sh:

# calculate how many levels we should go up to get to the root directory
echo $1
count=$(echo $1 | grep -o / | wc -l)
path_to_project_root=""
i=-1
echo "count $count"
while [ "$i" -le "$count" ]; do
  path_to_project_root+="../"
  i=$(($i + 1))
done
echo pathToProjectRoot=$path_to_project_root > garden.env

workflow.garden.yaml:

kind: Workflow
name: deploy-services

steps:
  - name: initialize
    description: initialize a service
    script: | 
      ./deployment-common/workflow-scripts/calculatePathToProjectRoot.sh ${local.projectPath}
  - name: deploy
    command: [deploy, "${replace(var.services, ' ', ',')}"]

template.garden.yml:

kind: ModuleTemplate
name: spring-service-module-template
inputsSchemaPath: rest.schema.json
modules:
  - type: exec
    name: ${parent.name}-module 
    ...
    generateFiles:
       # copy to .config folder
      - sourcePath: ${var.pathToProjectRoot}/${this.path}/${inputs.deploymentFolder}/config/common.yaml
        targetPath: ${inputs.deploymentFolder}/.config/common.yaml

Expected behavior

Garden should run the first step isolated from the second one.

Workaround

Either creating a separate workflow just to run the initialize step or creating a custom command (new garden feature) to call the workflow after running those commands.

Your environment

• OS: Ubuntu in wsl2
• How I'm running Kubernetes: GKE

garden version 0.12.34-edge-7fdd7022

[tedchang77]

Related to: https://github.com/garden-io/garden/issues/2703 and https://github.com/garden-io/garden/issues/2704

[thsig]

This is a result of the fact that workflows don't re-resolve configs between steps (which is done for performance reasons).

I think it would make sense to add a config option to specify that Garden should re-resolve configs for each step (or for specific steps).

That would slow down the workflow a little, but would enable the workflow you're describing there.

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[thsig]

This is still a valid use-case for certain scenarios, so let's keep this open for now.

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[vvagaytsev]

@carolinapc is this still an issue, or have you already found a good-enough workaround?

[carolinapc]

Hi @vvagaytsev, It's not an issue anymore, we have stopped using that script and garden custom commands are pretty useful! Thank you.

[vvagaytsev]

@carolinapc, thank you! Then we'll close this. Feel free to reopen if it becomes an issue again.