Passing multiple environments to helmfile

[dudicoco]

Hi,

I'm trying to pass multiple environments to helmfile:

helmfile -e staging -e prod build

This doesn't work as helmfile only takes the last -e argument.

Is there a way around this?

Thanks

[yurrriq]

Environments in helmfile are mutually exclusive. What is your intention here? Perhaps something like this:

environments:
  staging:
    values:
    - path/to/staging.yaml
  production:
    values:
    - path/to/staging.yaml
    - path/to/production.yaml

[dudicoco]

@yurrriq helmfile.yaml is unrelated to the helmfile cmd flags.

My intention is to be able to deploy to multiple environments with one command.

[dudicoco]

I've found a workaround - use an environment label instead of the native environment feature:

releases:
- name: amazing-app
  chart: my-repo/amazing-app
  version: 1.0.0
  installed: true
  namespace: devops
  labels:
    environment: staging
- name: amazing-app
  chart: my-repo/amazing-app
  version: 1.0.0
  installed: true
  namespace: devops
  labels:
    environment: prod

You can then deploy in parallel:

helmfile -l environment=staging,name=amazing-app -l environment=prod,name=amazing-app apply

My problem now is that I can't figure out which release belongs to which environment from the helmfile output, for example:

Upgrading release=amazing-app, chart=my-repo/amazing-app
Release "amazing-app" has been upgraded. Happy Helming!
NAME: amazing-app
LAST DEPLOYED: Tue Nov 30 15:31:16 2021
NAMESPACE: devops
STATUS: deployed
REVISION: 68
TEST SUITE: None

Listing releases matching ^amazing-app$
amazing-app	devops   	68      	2021-11-30 15:31:16.141045 +0200 IST	deployed	amazing-app-1.0.0 	1.0

Upgrading release=amazing-app, chart=my-repo/amazing-app
Release "amazing-app" has been upgraded. Happy Helming!
NAME: amazing-app
LAST DEPLOYED: Tue Nov 30 15:32:32 2021
NAMESPACE: devops
STATUS: deployed
REVISION: 2
TEST SUITE: None

Listing releases matching ^amazing-app$
amazing-app	devops   	2       	2021-11-30 15:32:32.92882 +0200 IST	deployed	amazing-app-1.0.0 	1.0


UPDATED RELEASES:
NAME          CHART                 VERSION
amazing-app   my-repo/amazing-app    1.0.0
amazing-app   my-repo/amazing-app    1.0.0

@mumoshu any suggestions?

[yurrriq]

If I understand your desired outcome correctly, labels are probably what you want, rather than environments. The output of helmfile list shows labels, which may be of some use.

If you're going to eschew helmfile's environments in favor of a label-based parallel deployment, perhaps it would be good to use unique release names, e.g. amazing-app-staging and amazing-app-prod, to help distinguish them.

[dudicoco]

Thanks @yurrriq.

I've thought about unique release names before, but it's not really an option because we have hundreds of existing releases and renaming them would be a breaking change.

Just tried helmfile list now, i'm not sure how could it help me figure out the output of helmfile apply?

[yurrriq]

helmfile list merely prints the labels attached to a particular release, so it won't be helpful with respect to helmfile apply, unfortunately. Is it untenable to (whether using helmfile's environments or not) have sequential calls to helmfile apply? If so, I'm out of ideas.

[clemcvlcs]

Hi guys, I also have the same need. I'm not sure to understand how helmfile list can help here. Is there a way to apply on multiple environments in one command line without having to create one release per environement, with a specific label for each one ? If not, it would be really great to add a feature like helmfile -e staging,prod sync

[renemifsud]

Hi guys, is it by design to not have an "all" option, because this would be a great feature for faster Disaster Recovery. Thanks

[mumoshu]

@clemcvlcs @renemifsud Hey! I hear you, but can't you just write one-line bash script to for-each helmfile run per environment?

[renemifsud]

@mumoshu, True but its not that much of a one liner, you have to have a reference of what the environment names are, you can get these by either parsing the helmfile.yaml file or create a separate file with the names in it and loop trough it

[mumoshu]

@renemifsud Just wondering, but what's your real issue? In your environment, folks other than you write many helmfile.yaml with a lot of environments, that makes it impossible for you to know which environments are defined in a helmfile.yaml beforehand?

[renemifsud]

@mumoshu The real issue is thats its better to have a nicer way than hacking around it. To make it generic you have to parse the helmfile.yaml on runtime and get the list of environments which doesnt make the shell script that straight forward. Helmfile is really useful because you can group all different releases of a chart in the same repo, having a functionality to update all the releases at once (since changes can be common for each release) makes Helmfile more powerful and useful in my honest opinion.

[mumoshu]

To make it generic you have to parse the helmfile.yaml on runtime and get the list of environments which doesnt make the shell script that straight forward.

Thanks for clarifying! What I'm concerned about is this point. What I mean by your "real issue" is the exact situation or goal of your project. If it's just a "nice-to-have" then it's not a good idea to implement it in helmfile first.

Let's say, for example, you have only two or three big top-level helmfile.yaml to work on. You can hard-code the environments, or if you have more time you can externalize the list of environment definitions to other files outside of the helmfile.yaml. You can then let helmfile.yaml read the environments files(perhaps they will be yaml files). helmfile already provides template functions like readFile and parseYaml that should be useful for such usecases. The end result is that you can read and parse the environments.yaml file directly from your script, without being forced to parse helmfile.yaml. If that works, enhancing helmfile -e to just avoid using readFile and parseYaml from within your helmfile.yaml seems to be overkill and might be insufficient.

[renemifsud]

Thanks for the fast reply @mumoshu Unfortunately, I don't agree. with you that this is an overkill or that it's just a way to avoid using readFile and parseYaml. You just gave me another way to do the same thing that I already do and the only thing I'm gaining from this is that I can use gotmpl to forloop the environments part of the helmfile.yaml using parseYaml from a seperate file, which still forces me to have a script to apply or deploy changes on all the releases. If I understand correctly what the scope if Helmfile is, correct me if I'm wrong, is that one can define multiple charts for multiple environments at once, so it makes total sense in my head that Helmfile should have also the functionality to update all releases at once. If I'm still forced to have a custom script to do this, I still consider it a dirty workaround, because with that argument I should have just built a script that templates the values of each environment using just Helm.

[mumoshu]

Hey! Thanks for clarifying. I think I almost got your position, but still unsure why it sounds like all-or-nothing.

If I'm still forced to have a custom script to do this, I still consider it a dirty workaround, because with that argument I should have just built a script that templates the values of each environment using just Helm.

If your goal can be achieved by building your own script that generates various values.yaml files and runs Helm, that might imply you don't need various advanced features of Helmfile. Then, yes, it's definitely an option.

I do use Helmfile myself because it provides many features I don't want to repeat implementing. So, If it's very rare to run Helmfile against all the environments at once, I'd just write a shell script that wraps helmfile as a starter, instead of throwing away Helmfile as not perfect.

Helmfile doesn't provide a lot of "hook" points so it's very likely we need to enhance Helmfile for anything that works per release, like the kustomize support Helmfile has today. But if you can do something "externally", there's less necessity to implement it in Helmfile- writing a short shell script is definitely an option in such a case. I think this feature falls into this category.

If I understand correctly what the scope if Helmfile is, correct me if I'm wrong, is that one can define multiple charts for multiple environments at once,

I agree with this.

so it makes total sense in my head that Helmfile should have also the functionality to update all releases at once.

Yes. Every helmfile command works like that. It's just that helmfile works against one environment at once.

I'd say my expectation for Helmfile environment is very similar to Terraform workspace. There's a "default" environment and if you run helmfile without -e, it's run against the default environment. You can definitely define multiple environments but you can select only one of those at runtime. That's how it works today.

I can imagine cases like you constantly work against multiple environments. That's why I asked https://github.com/roboll/helmfile/issues/1999#issuecomment-1082769885. With that question, I wanted real numbers, like the number of environments and helmfile.yaml files you need to manage. I thought I'd agree with your idea of implementing it in Helmfile if we had dozens of projects each has its own helmfile.yaml that requires the "disaster recovery" thing.

But you didn't give me numbers so I'm not yet sure if you are the first Helmfile user who works against dozens of projects while having the requirement for the disaster recovery across multi envs, or you're just saying it to avoid writing a Helmfile wrapper at all.

[mumoshu]

And I don't think enhancing -e to take multiple envs is practical alone.

If I were you, I'd want helmfile -e a,b,c (diff|apply|sync|template) to segregate logs per environment, and helmfile -e a,b,c list to segregate outputs per environment. Also, it's not entirely sure if each environment needs to be processed concurrently or not. If it needs to be processed concurrently, we'd need to prefix logs from each environment appropriately, otherwise we can't see which logs are from which environment. Another concern- what should happen if the first environment failed? Should Helmfile continue to second and third environments anyway, or fail-fast after the first environment? Does such behavior need to be configurable?

If we naively enhanced -e to take multiple envs, any of those concerns aren't addressed.