Filter out images at ImageRepository level

[squaremo]

In a given ImagePolicy object you may want to select a subset of the images, e.g., those marked as destined for a particular environment dev-*. You might also want to filter in an ImageRepository which images are available to policies; for example, if there are images that are known to be faulty, or cause problems for scanning.

I don't think this needs a big up front design, but it might pay to think (then write) about

• the "little language" for describing what to exclude or include
• how it fits in with filtering in image policies.

[relu]

Adding my comment from #58 here:

An extra thought on the policy level prefix matching feature. I think it would be best if we implement this after we integrate a "real" database into the controller so that we can use its capabilities to filter out the tags dataset.

[relu]

I've been thinking some more about this and it might not be a good idea to have this capability at the level of the repository, but rather more generic (as in not specific to a particular policy type) at the level of an image policy resource. The way I see things is that users can probably have a single repository resource with multiple policies responsible for different stages of a product's release lifecycle, some policies would match against a particular set of tags, some another (e.g. rc-<timestamp> for staging deployments and rel-<timestamp> for production).

[TonyLovesDevOps]

Could one use this to approximate the glob: tag matching functionality of fluxv1?

For example, our CI system generates docker images and tags them with the pattern <branch name>-<build number>-<git commit sha> e.g. a build on develop might look like develop-199-dae1ee8 and the next one develop-200-7aeb34c. We've been using this convention with fluxv1 and it works well with the annotation flux.weave.works/tag.foo-service: glob:develop-* in our deployments. A different branch might produce an image tag awesomeFeature-201-2bbd164 and this tag is ignored by flux since it doesn't match the glob.

It seems like fluxv2 is pushing for semver-compliant tags. I was initially thinking of just using something like 1.0.<build number>+<branch name>-<git commit sha> but then this breaks our workflow since (using the example above) 1.0.201+awesomeFeature-2bbd163 gets deployed to our cluster, when we want the latest image created from the develop branch.

Perhaps I'm missing something obvious, and am open to suggestions for how to achieve our desired outcome: for a given deployment, deploy the latest image tag generated from a given branch. Perhaps what I'm looking for is #15?

Also, am I right in deducing that annotations on k8s deployments are going away in fluxv2 i.e. there's no way within a deployment manifest file (and only a deployment manifest file and not an ImagePolicy) to define whether/how flux will perform automated deployments when a new docker image tag is pushed?

[relu]

Thanks for your suggestions, @TonyLovesDevOps. Flux2 actually does have support for an Alphabetical ordering policy that you can rely on, however, it does not yet support tag globbing (or prefix matching).

A potential example of how you can define such a policy with your use case would be:

apiVersion: image.toolkit.fluxcd.io/v1alpha1
kind: ImagePolicy
metadata:
  name: podinfo
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: podinfo
  policy:
    alphabetical:
      order: asc
    # note that this part is not yet implemented
    matchTags:
      prefix: develop-

Also, am I right in deducing that annotations on k8s deployments are going away in fluxv2 i.e. there's no way within a deployment manifest file (and only a deployment manifest file and not an ImagePolicy) to define whether/how flux will perform automated deployments when a new docker image tag is pushed?

Yes, this is correct. The automation logic is currently implemented in a dedicated controller and is independent of the workloads running inside the cluster. It relies on markers defined in your manifests (either workload resources or any CRD) e.g. # {"$imagepolicy": "flux-system:podinfo:tag"}. To be able to define an automation workflow you would need to create a resource using the ImageUpdateAutomation, detailed here.

[TonyLovesDevOps]

Thanks for the detailed response. I think we'll hold off on upgrading to Flux2 until more of the image tag matching is ironed out and implemented. Also, (and I know an only-tangentially-related Github issue comment is not the place to effectively share this kind of feedback but I'm not sure where else to do it), as an end user simply trying to "upgrade" from Flux/helm-operator v1 to the whole GOTK ecosystem is overwhelmingly complex. All I want is for my manifests/helm charts in git to be synchronized to my clusters, and my docker image pushes to be deployed automatically and synchronized back to git. Yes, GOTK can do it, but it seems like using an (admittedly awesome) aircraft carrier to kill a housefly.

Or, to illustrate the point using maybe my favorite animated GIF of all time, just replace "systemd" with "GOTK" and all of its snacks with "kubectl", "kustomize", "git", "docker", etc.

[squaremo]

That is an excellent animated gif :-D

Is it fair to paraphrase your feedback as "It's difficult to know where to start with Flux v2 because there are many more parts to it"? I wonder if we could have a simple menu of "If you use Flux v1 for X then the equivalent is Y (go here to see how to set it up)", so people coming from Flux v1 have obvious signposts into the relevant docs.

[stefanprodan]

I wonder if we could have a simple menu of "If you use Flux v1 for X then the equivalent is Y (go here to see how to set it up)", so people coming from Flux v1 have obvious signposts into the relevant docs.

Such a doc exists here: https://toolkit.fluxcd.io/guides/flux-v1-migration/#in-place-migration