Possibility to update remote git sources of sub-helmfiles

[dennybaa]

Hello @mumoshu!

It would be nice to be able to update remote git sub-helmfiles. I've suddenly found this thing for myself and already envision some useful scenarios. It would be nice to have an option/arg or command to enable "git sync" mode (to pull the changes), smth like bellow:

helmfiles:
- # Terraform-module-like URL for importing a remote directory and use a file in it as a nested-state file
  # The nested-state file is locally checked-out along with the remote directory containing it.
  # Therefore all the local paths in the file are resolved relative to the file
  path: git::https://github.com/cloudposse/helmfiles.git@releases/kiam.yaml?ref=0.40.0
  sync: true

Again any possibility which will be considered the best, even an option under helmDefaults (which is unfortunately used for helm specific options)...

[mumoshu]

envision some useful scenarios

@dennybaa Hey! I'm interested- What are the useful scenarios?

Helmfile syncs and caches the repository per ref so as long as you update ref=0.40.0 part to e.g. ref=0.41.0 it should sync the new remote content into helmfile's cache. Isn't that enough?

Or did you find it not working as I've described?

[mumoshu]

@dennybaa Just a friendly reminder 😃

[dennybaa]

Hey @mumoshu, sorry for the delay. Well, speaking about sync, I envision branch based scenarios. For example let's say we have some remote git repo similar to cloudposse/helmfiles and we might have tags fact or we might want to work with just the master branch (which is considered stable).

This approach will allow to centralize some common helmfiles and easily upgrade dependent projects. Yes might sound dangerous, but it's definitely can be useful during the development to pull the latest updates. Anyways there will be a choice to work with tags or use branches :)

[mumoshu]

@dennybaa Hey! Thanks. But I'm pretty lost. Are you trying to run helmfile from within some CI pipeline and you want to always pull the head of the branch denoted by the ref parameter?

Is that an equivalent to just running rm -rf .helmfile/cache && helmfile apply?

[dennybaa]

@mumoshu In a pipeline it can be mitigated by simply running without caching, so generally rm -rf is not needed. During development however it would be nice to avoid -rm. In my practice I use not only CI workflows with helmfile, but for a number of projects helmfile always invoked manually. For such projects in my opinion it would be nice to see an option, so when helmfile is invoked I could always get the "fresh upstream helmfiles"...

So this it, what I see it for only... More specifically when I'm talking about this "central/upstream" helmfiles repository, I have an idea to abstract from the actual chart values and try to provide a limited helmfile values interface which won't change much even if the underlaying chart which helmfile deploys changes.

[mumoshu]

@dennybaa Hey! I now see what you envision. So, you would usually use it to pull changes for remote helmfiles whose refs are like ?ref=main, but not ?ref=0.40.0, right?

One concern I have right now is that how would you differentiate/select which remote helmfiles to update, as there might be multiple helmfiles where one has ?ref=main and another has ?ref=0.4.0. You'll probably want to update the first, but not the second?

[dennybaa]

@mumoshu Hey! This is good point, how... I think the most straightforward way could be to encode this into the path, WDYT?:

path: git::https://github.com/cloudposse/helmfiles.git@releases/kiam.yaml?ref=main&sync=true

Also check the ref is actually a tag when sync=true then fire an error and exit. Simple without any extra yaml keys introduced.

[mumoshu]

@dennybaa Hey! Thanks for the reply. Could you help me shape this a bit more?

So- can we just re-fetch the git branch whenever you have sync=true? Or would you like to enable re-fetching only when you have sync=true AND you also have some command-line flag to enable it?

Maybe that's due to my limited understanding of your goal, but it's still unclear to me which behavior you'd prefer.

[dennybaa]

@mumoshu Hi there. Thanks for paying attention to these all. Well I'm starting to doubt that it can be easily implemented.

The bellow case makes me think that it won't be trivial...

helmfiles:
- path: git::https://github.com/dennybaa/hfr.git@releases/test/helmfile.yaml?ref=master
  values:
    - release: foo
- path: git::https://github.com/dennybaa/hfr.git@releases/test/helmfile.yaml?ref=master&sync=true
  values:
    - release: sync
- path: git::https://github.com/dennybaa/hfr.git@releases/test/helmfile.yaml?ref=v0.0.2
  values:
    - release: bar

So a global arg to helmfile to invalidate cache, i.e. just rm -rf cache would be the most straightforward way to do. But thinking even further about the recursive nature of the helmfile include process :) Maybe it makes sense to leave it as it is, unless it's possible to distinct the most parent process where to perform the cleanup...

[FWest98]

I ran into this as well just now - I did not realize the references were not being updated and I was wondering why the linter kept complaining about outdated versioning...

Maybe I'm missing something, but why would we not want to update the cache in some scenario? We could just always update the cache, right? ?ref=master and similar might actually get an update, but things like ?ref=v0.4.0 is intended to be fixed in time (until you change the tag in the repo) anyway, so it shouldn't matter?

I would even make the case that we would want to always update the cache; otherwise, the same code base might yield different results on different machines that have a different cache... That would be undesirable I think.

[jess-sol]

Running into this too. We plan on using ?ref=main, but I was hoping it'd record the sha of the reference into the lockfile, and something like helmfile deps would update it in the lockfile.

[bessny5-cell]

helmfiles:

• path: git::https://github.com/dennybaa/hfr.git@releases/test/helmfile.yaml?ref=1000 values:
  • release: foo
• path: git::https://github.com/dennybaa/hfr.git@releases/test/helmfile.yaml?ref=master&sync=true values:
  • release: sync
• path: git::https://github.com/dennybaa/hfr.git@releases/test/helmfile.yaml?ref=v0.0.2 values:
  • release: bar So a global arg to helmfile to invalidate cache, i.e. jus t =0628347846

[bessny5-cell]

helmfiles:

• path: git::https://github.com/dennybaa/hfr.git@releases/test/helmfile.yaml?ref=master values:
  • release: foo
• path: git::https://github.com/dennybaa/hfr.git@releases/test/helmfile.yaml?ref=master&sync=true values:
  • release: sync
• path: git::https://github.com/dennybaa/hfr.git@releases/test/helmfile.yaml?ref=v0.0.2 values:
  • release: bar So a global arg to helmfile to invalidate cache, i.e. just =0628347846