feat: Synchronize changes back to Git

[jannfis]

Is your feature request related to a problem? Please describe.

We need to find a proper way how we can synchronize our changes back to Git. Currently, we leverage Argo CD API to set the application parameters for image override, and Argo CD will update the Application resource with parameter overrides in the Kubernetes cluster. These changes need to be persisted somehow.

Describe the solution you'd like

I think the details about this need to be discussed and designed properly. This issue is open to collect all kind of ideas & discussions.

[jannfis]

There's some ongoing discussion in https://github.com/argoproj/argo-cd/pull/4084

[hawksight]

So in my setup, I'm using a directory inside a particular repo to define all my argocd projects / applications in yaml. General pattern is to update the application yaml in that subdirectory tree in a PR and have that reviewed / merged to master.

applications/
  <argocd-project>/<namespace>/<application>.yaml
  stg-eu/default/track.yaml

The argocd project == an environment == a k8s cluster. (A declarative setup.)

So my use case would be to reflect the auto image updates back to this single git repository but into the correct application crd yaml file. The feeling I get is that for the general community on slack, they all use a similar 1 repo setup, so perhaps the first iteration of this feature could just target this use case?

Some overall thoughts I've had:

• Allow a default global git-secret for the image-updater.
• Also use an annotation to override the secret for a given application argocd-image-updater.argoproj.io/git-secret: <my-specific-secret> - probably limited to secrets that reside where the image-updater is running (not other clusters)
• Some method of defining how to writeback the change. Branch and PR the change, but chose whether it's auto merged or left for review. This probably gets tricky with permissions, depending on how the credentials to git are setup (permissions). Maybe first time round, just create a PR and log in the image-updater.
• If someone rolls back an auto update, there should be something to go and close the PR or update it to say it was rolled back.
• Perhaps auto merge depending on whether auto-sync is enabled. If enabled, auto merge PR. If not, its needs approval. That would make sense matching git to reality.
• Some way to scope which file, either an application crd.yaml or maybe a helm values.yaml if people are doing that. Or will image updater always create a parameter that overrides meaning it only ever needs to edit an application crd yaml file?
• As well as opening a PR, show a diff of the changes in the message, so it obvious to humans what changed.
• Specify a commit message / pr title format, or be able to construct it somehow?

I followed the discussion here and liked the suggested annotations (if i understand them correctly).

Maybe some additions based on some of the points above:

annotations:
  # store parameters in crd
  argocd-image-updater.argoproj.io/writeback: crd
  # update parameters in app manifest file
  argocd-image-updater.argoproj.io/writeback: app-manifest-file:./apps/guestbook-app.yaml
  # parameters file
  argocd-image-updater.argoproj.io/writeback: parameters-file
  # Specific secret to use for git: <namespace>/<secret>
  argocd-image-updater.argoproj.io/git-secret: argocd-image-update/git-update
  # Chose whether to auto merge PRs or not. Default will match argcocd sync policy? This will override.
  argocd-image-updater.argoproj.io/git-auto-merge: false

[hobbsh]

The helm pattern should also be considered, specifically the chart dependency model (example here). I think this should be fairly similar to the approach @hawksight mentioned above, just wanted to call out the necessity for more flexibility in the annotations.

[jannfis]

Thanks a lot for some great suggestions and writing down your requirements, @hawksight and sorry for coming back so late to this one. But good news is, with https://github.com/argoproj/argo-cd/pull/5038 now merged into Argo CD, I felt like picking up the work on Git write-back in Image Updater again.

The general pattern now is, that the Image Updater will marshal an application specific source parameters file (i.e. .argocd-source-<application>.yaml) and commits it directly into the application's source repository, at least in the initial iteration. This will minimize the impact on the repository and also the possibility for merge conflicts would be close to zero, as long as no other party will modify the same file (which there is little reason to). Also, this will work regardless whether the application manifest lives in Git (e.g. in app-of-apps setups) or is created purely imperatively, i.e. using UI or CLI.

Some overall thoughts I've had:

* Allow a default global `git-secret` for the image-updater.

That's a great idea. I didn't think about this, but it's definitely a very useful feature. Will include a feature to specify a default secret.

* Also use an annotation to override the secret for a given application `argocd-image-updater.argoproj.io/git-secret: <my-specific-secret>` - probably limited to secrets that reside where the image-updater is running (not other clusters)

The way it is currently implemented it is to be able to specify secret as namespace/secret. The fields in the secret are currently hard-wired as having to be username and password for HTTPS repositories, and sshPrivateKey for SSH repositories. Would you think there'd be a benefit of having the fields configurable?

* Some method of defining how to writeback the change. Branch and PR the change, but chose whether it's auto merged or left for review. This probably gets tricky with permissions, depending on how the credentials to git are setup (permissions). Maybe first time round, just create a PR and log in the image-updater.
* If someone rolls back an auto update, there should be something to go and close the PR or update it to say it was rolled back.

* Perhaps auto merge depending on whether auto-sync is enabled. If enabled, auto merge PR. If not, its needs approval. That would make sense matching git to reality.

The initial feature will only be able to directly commit to a (definable) branch and push it to the remote Git repository. If that's the same branch being tracked by the application, Argo CD will pick up the change and potentially auto-sync to it. I feel creating a PR is something very provider specific and I'm unsure whether there's big demand for Image Updater creating PRs. But let's create another issue for this feature once the current feature is merged into Image Updater :)

* Specify a commit message / pr title format, or be able to construct it somehow?

This will be useful! Currently, the commit message is hard coded as well. In case of construction, I could think of a dynamic template, with variables being replaced with actual content. What do you think would be the most useful/valuable information here?

[hawksight]

The fields in the secret are currently hard-wired as having to be username and password for HTTPS repositories, and sshPrivateKey for SSH repositories. Would you think there'd be a benefit of having the fields configurable?

As long as the key names are documented then I don't see any benefit in having them configurable. These seem like good choices at the moment :+1:

This will be useful! Currently, the commit message is hard coded as well. In case of construction, I could think of a dynamic template, with variables being replaced with actual content. What do you think would be the most useful/valuable information here?

I like the idea, I'm just not sue what content. Probably easier to see in action and then work out what is required. Application name, image / tag updated and when the image was published?

I feel creating a PR is something very provider specific and I'm unsure whether there's big demand for Image Updater creating PRs. But let's create another issue for this feature once the current feature is merged into Image Updater :)

I'll raise one in the future if I still think its necessary.

To give you my reasoning.. auto image updates is useful and handy... but in many cases you want the image updater to do the hard work of telling you there is a new image to deploy, but not actually deploy it.

We have one repo with master branch defining all our environments (app of apps). Ideally, once some application CI produces a new image, i'd like:

1. Image-updater notices and evaluates against rules
2. Creates a commit on a branch in our repo
3. PR or MR (if using GitLab) from the new branch to master
4. That PR is then either auto-merged if allowed, or ready for the developers to merge... if it needs reviewing. As part of that PR it'd be really useful to have the expected changes in git (the argoCD diff). This way it can be reviewed directly in git without having to go to the ArgoCD UI.
5. Once merged, ArgoCD picks up the change and makes it happen in the same way it currently does

Essentially if changing our infrastructure code base, we like to review things. It also gives us some control over when things go out... but with the benefit of not having to make the PR or commit.

If say someone was adding some extra variables to the application they would have to follow the same process:

1. create a branch and make changes
2. PR changes to the master branch
3. Have someone review the changes and merge PR
4. ArgoCD picks up change to master and makes deployment.

I guess what I'm looking for is some way of having a consistent process for all changes, so image updates are handled the same way as any other application changes. This is the reasoning behind the use of PR's... it's the review / approval stuff. I think I could do some of this with manual sync in ArogCD UI instead of auto syncing. But git should be the source of truth for our infrastructure, so we usually track deployments or updates via the PR's merged.. this is why I think it makes sense to use PR's or MR's. Maybe we can get the same just be viewing commits. Or perhaps we can just have it set to update a non master branch and manually PR changes from that branch.

Sorry for another long response! Probably a case for me to go update to latest version and do some experimenting with the workflow options.

[an-tex]

I'd also love to see PRs being created. The approach could be similar to the brilliant ScalaSteward project, which creates PRs when there are library dependency updates.

In this case support for both, GitHub PRs and GitLab MRs would be great.

[joebowbeer]

I came here looking for an update on image-updater creating PRs. PRs are required when deploying from protected branches. A lot of devs are having to cobble this into their CI/CD, but it would be better if image-updater could do it.

A @jannfis comment mentions creating a separate issue for this feature, but I don't see one.

[abrenk]

The fields in the secret are currently hard-wired as having to be username and password for HTTPS repositories, and sshPrivateKey for SSH repositories. Would you think there'd be a benefit of having the fields configurable?

As long as the key names are documented then I don't see any benefit in having them configurable. These seem like good choices at the moment +1

Kubernetes Secrets have built-in types for that:

• kubernetes.io/ssh-auth
  • ssh-privatekey (different spelling than "sshPrivateKey"!)
• kubernetes.io/basic-auth
  • username
  • password

[rakeshramakrishnan-hbk]

commits it directly into the application's source repository, at least in the initial iteration.

Also, this will work regardless whether the application manifest lives in Git (e.g. in app-of-apps setups) or is created purely imperatively, i.e. using UI or CLI.

For the app-of-apps setup, how to setup write-back to the git repo which contains the application manifests, as opposed to the source repo (helm-chart) of the application?