Propagate resources across all Tenant Namespace using the TenantResource custom resource

[prometherion]

trafficstars

Current design

Alice would like to create several resources on each of their Namespaces. Upon the creation of each Namespace, they have to create the desired resources on each Namespace. Any automation cannot be put in place here.

Desired design

A new CRD could be crafted as follow.

apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
  name: database
  namespace: solar-management
spec:
  resyncPeriod: 60s
  additionalResources:
    namespaceSelector:
      matchLabels:
        tier: one
    items:
    - apiVersion: presslabs.io/v1beta1
      kind: MySQL
      spec:
        foo: bar
  clusterRoles:
    namespaceSelector:
      matchLabels:
        tier: one
    items:
      - name: database-admin
        subjets:
        - kind: Group
          name: developers

The Namespace-scoped resource TenantResource would allow alice to create on each Tenant namespace resource declared in the /spec/additionalResources key.

These resources (like a DB CRD) would require RBAC to developers, granted using the /spec/clusterRoles specification key.

Replication of these resources could be furthermore fine-grained using a namespaceSelector, allowing to define on which Namespace these should be replicated.

A reconciliation to ensure the selected status is expected is controlled by the resyncPeriod, a time-based value that allows to poll the current state of the resources and ensure the desired one.

This new custom resource definition is going to be defined as Namespace-scoped in order to allow Tenant owners to create their resources in a fashion automated way.

Proposal contributors

• @MaxFedotov
• @prometherion
• @carpenterm

Looking for contributors for this feature!

[slushysnowman]

Looks pretty good - would this also allow specifying pre-existing resources to be created in namespaces? Or would the spec of the item to be created in the namespaces have to be defined within TenantResource?

I'm imagining this could also be used to solve this issue? https://github.com/clastix/capsule/issues/222 - my only concern is if we specify secrets on this spec (although maybe we can use a CRD to retrieve them from elsewhere)

[oliverbaehler]

Sure thing, that's a great proposal and would certainly help with #416.

For the selector it would be awesome if we could also use matchExpressions instead of only namespaceSelector. This would allow more generic assignments, eg:

apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
  name: database
  namespace: solar-management
spec:
  clusterRoles:
    namespaceSelector:
      matchExpressions:
        - key: "production"
          operator: Exists
    items:
      - name: production-view-only
        subjets:
        - kind: Group
          name: developers

Sorry if it's already implied with the namespaceSelector that this should work as well.

[prometherion]

For the selector it would be awesome if we could also use matchExpressions instead of only namespaceSelector. This would allow more generic assignments

@oliverbaehler we're going to support it using the metav1.LabelSelector, so yes, end-user will be able to pick the preferred selection of items.

Looks pretty good - would this also allow specifying pre-existing resources to be created in namespaces? Or would the spec of the item to be created in the namespaces have to be defined within TenantResource?

Referring to an external object could be tricky, mostly due to a precondition regarding its existence: what should we do if the referred object doesn't exist? Should we block reconciliation, or just log it out or broadcast event about it?

Let's start the discussion about it: from the simplicity standpoint, referring bare resources in the TenantResource manifest is the best choice; on the other side, having the chance to specify external resources would provide more flexibility.

We could mix the two scopes: if the object in the items array is a k8s.io/api/core/v1/ObjectReference, we'll resolve it and render it.

Could it work, @slushysnowman?

[bsctl]

@prometherion even if it is a good and smart design, I’m having some concerns about this approach. Who’s in charge of creating the TenantResource object? If we leave the tenant owner the duty of creating additional resources in such way, we’re breaking the Capsule rule of native Kubernetes experience since the tenant owners have to create custom resources specific of Capsule as they were forced to do with OpenShift or Rancher. WDYT?

[oliverbaehler]

@bsctl @prometherion Have you had further discussions about this feature?

[prometherion]

@oliverbaehler my bad, missed this notification.

Since #644, we're planning this for the v0.2.0 release, and landing in the v1beta2 Tenant specification, too.

[prometherion]

@prometherion even if it is a good and smart design, I’m having some concerns about this approach. Who’s in charge of creating the TenantResource object?

The idea is to support both Namespaced resources, so Tenant Owner, if enabled by RBAC, can create their own replication resources, and in the Tenant specification for v1beta2.

[prometherion]

After a chat with the maintainer, we decided to go in a direction where two new CRDs are introduced, a cluster-scoped one named GlobalTenantResource, and a Namespaced one, TenantResources.

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
  name: green-production
spec:
  tenantSelector:
    matchLabels:
      energy: green
  resyncPeriod: 60s
  pruningOnDelete: true
  resources:
    - namespaceSelector:
        matchLabels:
          environment: production
      additionalMetadata:
        labels:
          labels.energy.io: green
        annotations:
          annotations.energy.io: green
      namespacedItems:
        - apiVersion: v1
          kind: Secret
          namespace: default
          selector:
            matchLabels:
              replicate: green
      rawItems:
        - apiVersion: v1
          kind: Secret
          metadata:
            name: raw-secret-1
        - apiVersion: v1
          kind: Secret
          metadata:
            name: raw-secret-2
        - apiVersion: v1
          kind: Secret
          metadata:
            name: raw-secret-3

apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
  name: wind-objects
spec:
  resyncPeriod: 60s
  pruningOnDelete: true
  resources:
    - namespaceSelector:
        matchLabels:
          environment: production
      additionalMetadata:
        labels:
          labels.energy.io: wind
        annotations:
          annotations.energy.io: wind
      namespacedItems:
        - apiVersion: v1
          kind: Secret
          namespace: wind-production
          selector:
            matchLabels:
              replicate: solar
      rawItems:
        - apiVersion: v1
          kind: Secret
          metadata:
            name: wind-secret-1
        - apiVersion: v1
          kind: Secret
          metadata:
            name: wind-secret-2
        - apiVersion: v1
          kind: Secret
          metadata:
            name: wind-secret-3

The last PR attached to this issue contains all the required information, along with the CRDs documentation.