Define CRD for Support Bundle kinds for inclusion with app

[markpundsack]

Define CRD for Preflight and Support Bundle kinds such that they can be installed with an application, and then kubectl support-bundle -n <namespace> could detect the support bundle spec(s) defined in the namespace rather than having to look at some out-of-band location for them.

Maybe kubectl support-bundle without a namespace should look for all support bundle definitions in the entire cluster, give the user a choice which one to run, or even to run all of them?

Note: Kots bundles specs with applications, but I don't believe they're installed as custom resources in the app namespace; they're stored in some other way.

[manavellamnimble]

@markpundsack Hi! I've create a beta CRD for support bundle. I want to have your opinion bf working further on it. Is this what you had in mind?

I used the current schema as validation for the CRD objects created. You can create a new support bundle object with the following yaml:

apiVersion: stable.replicated.com/v1
kind: SupportBundle
metadata:
  creationTimestamp: "2020-09-30T18:38:48Z"
  generation: 1
  name: new-support-bundle
  resourceVersion: "1738972"
  selfLink: /apis/stable.replicated.com/v1/supportbundles/new-support-bundle
  uid: 90cbc0e2-4bb5-47ad-aa3c-d877b5f15af4
spec:
  analyzers:
  - clusterVersion:
      checkName: Testing CRD
      outcomes:
        fail:
          message: Random fail message
          uri: randomuri

Running the command kubectl get sp new-support-bundle -o yaml you get:

apiVersion: stable.replicated.com/v1
kind: SupportBundle
metadata:
  creationTimestamp: "2020-09-30T18:38:48Z"
  generation: 1
  name: new-support-bundle
  resourceVersion: "1738972"
  selfLink: /apis/stable.replicated.com/v1/supportbundles/new-support-bundle
  uid: 90cbc0e2-4bb5-47ad-aa3c-d877b5f15af4
spec:
  analyzers:
  - clusterVersion:
      checkName: Testing CRD
      outcomes:
        fail:
          message: Random fail message
          uri: randomuri

[markpundsack]

Awesome, thanks! That's pretty much what I was thinking. Since then, we've introduced a new challenge, which is that support bundles can contain sensitive credentials. I wonder if we would need to encrypt this content. That might dramatically change things.

@divolgin is working on exposing (post-rendered) support bundles via k8s secrets or config items and I wanted to see if there was a way to have that be a step towards using a full CRD for them.

[manavellamnimble]

@markpundsack @divolgin It sounds interesting. I think K8s recommends for CRDs to use secrets for sensitive data. I can think in a couple of things:

• trying to use EncryptionConfiguration to encrypt the resource supportbundle or preflight at rest, but it might be tricky and I am not entirely sure it would work with resources other than secrets.
• encrypt secrets at rest for sensitive info and name them in the specs, although this might be tricky too, because we should introduce a KMS provider to make it really secure.

Should I complete the schemas for preflight and supportbundle resources? Or shall we discuss this further before creating the crds?

[markpundsack]

That's probably enough for now. We need a design proposal before forward.

[manavellamnimble]

@markpundsack Hi mark! I think the CRD's are already defined here. The CRDs there are similar to the ones I was working on, and allow the definition of kinds supportbundle, preflight and many others.

[manavellamnimble]

I could work on a new feature to parse the specs from a supportbundle or a preflight definition in the cluster.

[markpundsack]

I think the challenges are around the rest of the system. Like is there an operator that responds to these custom resources? How do we install the CRD when we have limited permissions? (CRDs require cluster admin, right?) Now that we potentially have secrets in the specs, how would we secure them if they're stored as custom resources (rather than encrypted config or secrets). How would the kots admin console and support bundle CLI make use of the custom resources? (the last part is probably the easiest.) How would vendors distribute these specs, include template rendering?

This would involve changes to vendor web, kotsadm, and troubleshoot.

[xavpaice]

@markpundsack recent work has allowed us to have multiple specs supplied to Troubleshoot, but at this point we have shied away from installing the CRDs in the cluster and storing specs as custom resources even though that's how the code is structured internally. I gather this is due to some permission issues which prevent some customers from using CRDs in this way.

We currently have the ability to search for secrets with a label which defines them as SupportBundle type specs, and we're working on extending that to configMaps for Redactors. This gets us to a similar result, but without the better control and searchability of CRDs that are installed.

Do we consider the CRD aspect of this issue/feature as still desirable?

[markpundsack]

CRD feels more natural as a user, but I agree the intent is satisfied with the current implementation so this can be closed.