Support remote manifest definitions

[AaronME]

What problem are you facing?

kubectl apply supports applying remote manifests, such as manifests available on github.

When an operator chooses to use an alpha or beta version of an operator, it is necessary to apply updated CRDs files which are very large and would clutter a composition.

How could Crossplane help solve your problem?

Add the ability to source a manifest from a URL instead of including it in-line.

[turkenh]

I believe this should be straightforward as long as we have 1 manifest per URL, which means there is still one-to-one mapping between an Object CR and the remote resource.

Otherwise, we will probably face some other issues related to managing multiple resources from a single managed resource.

[morningspace]

Agree w/ @turkenh, we can start from simple, and maybe can follow the similar pattern that provider-terraform uses.

[bobh66]

Hi @AaronME - How would you expect the reconciliation loop to handle a remote manifest? Does it pull the manifest once at Create time and only reconcile against that? Or does it pull the remote manifest on every reconciliation cycle (every minute) and reconcile against the current version of the remote file?

I am looking at a similar situation in provider-terraform which is currently retrieving and reconciling the remote terraform module on every cycle. I'm not sure that would scale very well in a large deployment, especially when it is reasonably unlikely that the remote content will change very often.

Thanks

[AaronME]

@bobh66 I believe the flow you an @negz discussed in the Crossplane Community meeting on 5/5 would work well. Cache a a local copy of the remote manifest within the container or in memory, and only pull if the pod is recreated. This would amount to a pull policy of "IfNotPresent."

It might be nice to support a flag similar to "imagePullPolicy" which allows the user to set one of three values: "Always," Never" or "IfNotPresent".

[ChrisJBurns]

This would be nice, as currently we are trying to figure out a way of installing Flux onto the clusters that we create in our compositions. One way is to possibly have a Crossplane provider for Flux or simply having the gotk-components.yaml file saved in Git and referencing it in the kubernetes provider.

Something like:

  - name: install-flux
    base:
      apiVersion: kubernetes.crossplane.io/v1alpha1
      kind: Object
      spec:
        forProvider:
          manifest:
            filePath: ../../gotk-components.yaml
...
...

[ChrisJBurns]

Or even so, we could just point it to the actual versioned gotk-components.yaml/ install.yaml on Flux official site. Instead of keeping a copy locally.

Although I can imagine in cases of airgapped networks (which occur a lot of Government for the obvious reasons), it may be useful to allow for someone to provider the kubernetes manifests by pointing to a local file that exists in the directory?

[bobh66]

The main limitation is the requirement that there can only be one Kubernetes object in the remote manifest. Any remote manifest with more than one YAML document in it would need to be treated as invalid, since the provider Object mapping is one-to-one not one-to-many.

[ChrisJBurns]

Ah yes, there is also that..

[bobh66]

Maybe the solution to that is a new "ObjectCollection" resource that handles the remote manifest and processes it's contents into Objects

[ChrisJBurns]

Otherwise, we will probably face some other issues related to managing multiple resources from a single managed resource.

I'm curious if we run into this if we go towards ObjectCollection. Or if that's endemic to Object.

[haarchri]

@ChrisJBurns
you can use flux2 helm-chart https://github.com/fluxcd-community/helm-charts/tree/main/charts/flux2 and flux2-sync helm-chart https://github.com/fluxcd-community/helm-charts/tree/main/charts/flux2-sync

with provider-helm

[ChrisJBurns]

@haarchri Are there examples of this being used in Crossplane that you could provide reference too?

[haarchri]

Check this closed PR https://github.com/upbound/platform-ref-aws/pull/60 we using it like this in our production environment

[ChrisJBurns]

@haarchri Are you doing anything with Deploy keys / ssh / knownhosts? Or are you configuring Flux to have https access to Git instead of ssh?

[haarchri]

You can mount all the stuff like ssh-keys, known-hosts, .netrc as Secrets / ConfigMaps - to source-controller - you can ping me as well on slack

[ChrisJBurns]

@haarchri Which Slack are you on? I'm on the kubernetes and cncf at the moment. I suppose the question is whether we can use the gitlab or github crossplane providers to generate the ssh keys and then use them to configure Flux with.

[morningspace]

@ChrisJBurns I am curious what the final resolution you picked in your case. It looks the discussion was about flux, but I think the original purpose of this issue is to ask for a way to apply massive k8s manifests using Object. Maybe we shall start w/ the one-to-one case, but I'm not sure if that's common enough, comparing w/ ObjectCollection. If ObjectCollection is more common, then we may need a proper design w/ some valid and concrete cases.

[ChrisJBurns]

Hi @morningspace

So I was floating a few ideas around. One of which - in this post I touched on - was installing Flux via it's manifests gotk-components.yaml and there's a lot of them, so having ObjectCollection would have been nice for that. But as @haarchri stated, there were better and more convenient ways of installing it, which is using the Flux helm charts. This was what I did in the end so I don't necessarily need the ObjectCollection anymore. However I still think it is a valuable thing to have for others who may want to apply multiple Object's.

[bobh66]

I think one issue may be that there is no way for the provider to know whether the remote manifest contains one object or many until it retrieves the manifest. For the Object case the provider could use the first YAML object that it finds in the manifest and ignore the rest, maybe with a Warning event to notify the user that it's not instantiating the entire manifest. The ObjectCollection case could handle 1 or more objects in the manifest. There is also the possibility that the content of the remote manifest could change from 1 object to more than one, which would not be reflected in the Object reconciliation.

This almost makes me think that ObjectCollection makes more sense for remote manifest handling since it can always reconcile whether there are 1 or many objects in the remote file.