[RFE] Allow a single backup to utilize more than one ResourceSet

[mallardduck]

Idea

Allow users to define backup jobs that utilize multiple ResourceSets.

Other changes/concerns:

• Adjust the Backup CRD to allow users to define multiple ResourceSets to use.
• Adjust backup controller to handle multiple ResourceSet backups
  • Consider: done as a loop, or ResourceSets merged before backup
• Shouldn't affect restore process
• Rancher GUI currently doesn't have any fields for this; we'd need to have some added.

Reasoning

When users customize a ResourceSet and start taking backups with it that they either: a) need to duplicate the Rancher default one, b) now have 2 backup files with data (1 with rancher default and 1 with their custom rule), or c) no longer backup rancher data.

User Experience

NOTE: This is a draft proposal and has not been approved, nor is it a design document either.

Users would never need to customize the Rancher ResourceSet or worry about messing up our official rules. Nor would they have to copy our Rules into their custom resource set which could be error prone and fall out of date fast.

Example of backup defined to use multiple sets:

apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
  name: multiple-resource-sets
spec:
  resourceSetNames:
  - rancher-resource-set
  - custom-user-set

Via the GUI: The user should have to take concerted effort to NOT use the Rancher backup resource set. I.e. It should default to include the Rancher one when they try to use multiple; then they can manually remove the rancher one if needed.

Alternatives

ResourceSet can define children set to merge in

We could continue to only accept a single ResourceSet and could instead update the ResourceSet to allow an "extra" set to be used in addition. The main difference here is that as Rancher O&B team, who publish the Rancher Backups default resource set, we could retain control over what additional sets can be used. The benefit of this greater control is that it provides a more restrictive framework for what customers can put into the Rancher backup.

This concept and perspective is important since BRO can be used as a general backup tool, however our main implementation of Rancher Backups still needs to optimize and consider Rancher Backups most. And by allowing us to control that when we release the chart, instead of giving users complete control at Backup creation time, can provide us greater ability to simplify the out of the box experiences.

Implementation Details

The new crd field: subResourceSet (or similar) is added as a string reference to another named resourceset. When BRO backup runs it will:

• See if the resourceset has a subResourceSet set,
• if so, it will:
  • try to look up a named resrouce matching that value,
  • if a match is found, it will:
    • fetch the subResourceSet rules,
    • merge the subResourceSet rules with the current sets rules
• continue to run the backup creation process as normal (but with potentially merged rules)