Proposal: add option for remote values files located in other repos

[aweis89]

Hey,

I think it would be awesome to have the option of pulling values files over http/s and git ssh. Something like:

values:
- https://some-url.com/master/values.yaml.gotmpl
- git://[email protected]:org/repo.git//branch/path/to/values.yaml
- ./local-values.yaml

Motivation:

We're using helmfile for our GitOps deploy workflow. We have repository of basically just helmfiles that specify the state for each env (like the docker image value and helm chart version). A change to said repo triggers a deploy to the corresponding environment where our ci basically just has to run helmfile sync in the correct directory. However we like to store values that are coupled with the application in the application repository which is separate from the helmfile repository.

If the above makes sense, we'd be happy to make a pull request for this.

Thanks

[mumoshu]

@aweis89 Hey! Yes, probably this makes sense. How will you implement it?

For more context, @AssafKatz3's #29 would help understanding why helmfile didn't add the feature in its early days.

You should also see requests for secret caching(#444) and vault integration(#392).

I was going to address those in a tool external to helmfile. However it turned out now to work well with a lot of "sub-helmfiles"(helmfiles: [...] in your helmfile.yaml). That is, you could write a helmfile-wrapper that fetches remote values/secrets to be consumed by helmfile. But it's time consuming when you have many sub-helmfiles, as basically the wrapper must initially fetch all the values/secrets required by all the sub-helmfiles.

[mumoshu]

Remote helmfiles has been implemented in #648. Adding values files support based on that work should be easy enough.

It would work as @aweis89 has initially proposed. One difference would be that helmfile uses @ for separation between (1)the dir to be checked-out and (2)the path to the values file to be loaded from it.

The another would be that helmfile uses, in the git provider for example, queries like ?ref=<tag or branch or commit id> for specifying what to check-out.

So git://[email protected]:org/repo.git//branch/path/to/values.yaml should be git://[email protected]:org/repo.git//path/[email protected]?ref=branch when you only want the files under path/to to be fetched along with values.yaml.

In case your values.yaml references to other files, e.g. via readFile, in upper levels, omit // from the left-hand side of path delimited by @, like git://[email protected]:org/repo.git@path/to/values.yaml?ref=branch.

[kamsz]

@mumoshu I'm trying to use this feature with SSH and I'm failing to do so. Example path you've said git://[email protected]:org/repo.git//path/[email protected]?ref=branch results in:

in ./dsf.yaml: in .helmfiles[0]: locate: get: error downloading 'git://github.com:org/repo.git?ref=branch': invalid port number "org"; if using the "scp-like" git address scheme where a colon introduces the path instead, remove the ssh:// portion and use just the git:: prefix

helmfiles:
  - path: git://[email protected]:org/repo.git//path/[email protected]?ref=branch

Using:

- path: git::ssh://[email protected]/[email protected]?ref=branch

actually DOES something, but it omits the git username.

[mumoshu]

@kamsz Yeah I observed it as well and it seemed like issues in go-getter that helmfile uses under the hood. That said, there's no easy fix.

What worked for me is use git::https://github.com/... for path (hence go-getter) and tell git to "convert it back" to git url... Confusing? Yeah but that what worked for me..

So you should have helmfiles like:

- path: git::https://github.com/yourorg/your-private-repo//pkg/whatever/[email protected]?ref=master

And place .gitconfig under your $HOME:

[url "[email protected]"]
        insteadOf = https://github.com/yourorg

[mumoshu]

Note that the above workaround is needed only for private git repos. Public git repos works without the workaround.

[kamsz]

@mumoshu If Terraform uses go-getter under the hood, it's pretty weird, because it works fine with TF.

[mumoshu]

@kamsz Interesting! Not sure what makes the difference.

Does your git url works when it is fed to go-getter https://github.com/hashicorp/go-getter ?

[mumoshu]

A few examples of mine:

git://[email protected]:ORG/REPO//DIR:

$ go-getter git://[email protected]:mumoshu/private-helmfile-repo-for-testing.git//dir?ref=master test1
2019/06/25 12:08:10 Error downloading: error downloading 'git://[email protected]:mumoshu/private-helmfile-repo-for-testing.git?ref=master': invalid port number "mumoshu"; if using the "scp-like" git address scheme where a colon introduces the path instead, remove the ssh:// portion and use just the git:: prefix

git://[email protected]/ORG/REPO//DIR:

$ go-getter git://[email protected]/mumoshu/private-helmfile-repo-for-testing.git//dir?ref=master test2
2019/06/25 12:11:16 Error downloading: error downloading 'git://[email protected]/mumoshu/private-helmfile-repo-for-testing.git?ref=master': /usr/bin/git exited with 128: Cloning into '/var/folders/_w/3lwtgvv51tl_fgxkdvcmtm340000gp/T/getter762174687/temp'...
fatal: unable to look up [email protected] (port 9418) (nodename nor servname provided, or not known)

git:ssh://[email protected]:ORG:REPO//DIR:

go-getter git::ssh://[email protected]:mumoshu/private-helmfile-repo-for-testing.git//dir?ref=master test1
2019/06/25 12:09:22 Error downloading: error downloading 'ssh://[email protected]:mumoshu/private-helmfile-repo-for-testing.git?ref=master': invalid port number "mumoshu"; if using the "scp-like" git address scheme where a colon introduces the path instead, remove the ssh:// portion and use just the git:: prefix

git:ssh://[email protected]/ORG/REPO//DIR:

$ go-getter git::ssh://[email protected]/mumoshu/private-helmfile-repo-for-testing.git//dir?ref=master test1
2019/06/25 12:10:17 success!
$ ls test1
helmfile.yaml

[mumoshu]

So your last example should work:

• path: git::ssh://[email protected]/[email protected]?ref=branch

Perhaps helmfile is dropping the git@ part while parsing the url? I'll take a look into it and fix it if needed.

[kamsz]

@mumoshu Yeah, it's dropping git@ part. I thought I mentioned that earlier:)

[mumoshu]

@kamsz True! Sry I was so dumb!

[mumoshu]

@kamsz #719 should fix that

[kamsz]

@mumoshu Thank you! Appreciate that:)

[mumoshu]

@kamsz Glad to help, and thanks again for reporting! FYI, I've just released v0.79.2 which includes the fix 😃

[kamsz]

@mumoshu If I may suggest one more thing - an argument to force cache refresh. After remote helmfiles got downloaded, they're not downloaded again until cache is removed.

[mumoshu]

@kamsz Worth another feature request but anyway - Yes that makes sense!

Do you expect U/X similar to #593, or perhaps just a helmile sub-command to refresh cache other than running rm -rf .helmfile/cache?

[kamsz]

@mumoshu One more question. Is the following syntax supported?

helmfiles:
  - path: git::ssh://[email protected]
    values:
      - values.yaml

From what I can see values are not being merged.

[mumoshu]

@kamsz Unfortunately no, but it worth a feature request.

I thought mentioned that config syntax in somewhere else, willing to add it to helmfile, but seems like never got to create a feature request myself!

[mumoshu]

Oh wait... I lost my memory

[mumoshu]

Sry it's actually implemented in #640 :) Are you sure you're explicitly referencing a key existing in values.yaml with {{.Values.whatever.key}} in the dsf.yaml?

[osterman]

@mumoshu has now made this possible with https://github.com/mumoshu/terraform-provider-helmfile

Use terraform remote state for “remote values” ;-)

[Manuel9924]

Can someone teach me how to go about it?

[Dustin77]

hg5twl5rpv47rjzo

[Dustin77]

hg5twl5rpv47rjzo

[XenoAura]

@mumoshu One more question. Is the following syntax supported?

helmfiles:
  - path: git::ssh://[email protected]
    values:
      - values.yaml

From what I can see values are not being merged.

I created an issue for this case https://github.com/roboll/helmfile/issues/1205

[johnmarcou]

The workdaround I am using for now, to load remote values to use with a remote helmfile module.

helmfiles:
  - path: git::ssh://[email protected]/company/[email protected]?ref=master
  - path: git::ssh://[email protected]/company/[email protected]?ref=master
    values:
      - .helmfile/cache/ssh_bitbucket_org_company_config.ref=master/config.yaml

The first helmfile module is a dummy one, no release in there, just config. But helmfile bin will still clone it locally. The second helmfile is the actual helmfile to render, but we can inject the local cached "remote" values.

[kevinjqiu]

Is there a way for the environment values file being loaded from a remote location?

The use case is that we put our helmfiles in the project repo, and there are some global environment config values we want to load into the helmfile.

something like:

environments:
  dev:
    values:
      - "git::https://mycompany/helmfiles/__global__/dev.yaml"

[kevinjqiu]

I took a first stab at implementing what I described above. Here's the PR: https://github.com/roboll/helmfile/pull/1296

Please let me know if this is alright. Thanks!

[alexku7]

I have additional suggestion for improving .

The ability to specify the chartmuseum as the source of custom values files. The scenario is similar to the original proposal of using the git for the storing the charts

As mentioned before, we are using the chartmuseum as the "source of truth" for each environment (dev, testing, staging, production) and not git. We have found it more useful and convenient .

On other hand the deployment could be executed from the project repo by ci/cd but also by the centrak GitOps mechanism (let's call it so) . So we need the option for specifying custom values for the environments but in order to keep the all deployments consistent we would like to read the values from one single source of true - the yaml files that are stored with the chart itself in the chart museum

[roanosullivan]

I took a first stab at implementing what I described above. Here's the PR: #1296

Please let me know if this is alright. Thanks!

@kevinjqiu It looks like this only works with environment values. Any chance your merged PR 1296 also included support for remote values files for releases?