feature: partitioned APIExport virtual workspace endpoints

[embik]

Feature Description

It might be desired to allow multiple entities to reconcile the same API (exported via an APIExport), for example in a marketplace setup where multiple providers support a generic KubernetesCluster API. Another use case for this would be running multiple api-syncagent instance across a fleet of clusters, with each of them supposed to reconcile a subset of objects of a particular resource.

To facilitate this, the APIExport virtual workspace should optionally allow partitioning of objects and offer dedicated endpoints for each partition. These endpoints can then be used by the different "providers" to reconcile just their objects.

Objects should be partitioned either by a label or by a particular field in the spec (think ingressClass or similar). To gain access to a particular partition, an extension of our existing RBAC model should allow granting access to a particular partition.

Proposed Solution

Some considerations for designing this feature:

• Do object partitions need to be static or can they be dynamic? i.e. do you need to configure just the key that creates partitions or also the possible values?
• Do object partitions include resources claimed via permission claims? This is likely connected to #3410 and the admission framework we need to put in place.

Depending on the solution, we might be able to extend APIExportEndpointSlice with an additional field:

kind: APIExportEndpointSlice
apiVersion: apis.kcp.io/v1alpha1
metadata:
    name: example
spec:
   objects:
     partitionedBy:
       label: provider.kcp.io
[...]

RBAC could look like this (unclear: how can we encode partition information in the resource/subresource pattern of RBAC?):

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: example:access-partition
rules:
- apiGroups: ["apis.kcp.io"]
  verbs: ["*"]
  resources:
  - apiexports/content-partition:provider-a
  resourceNames:
  - example

Binding this ClusterRole to someone would give this person access to the object partition that only includes objects with the provider.kcp.io=provider-a label.

Labels and fields used for partitioning should be considered protected and immutable. Once an object has been "scheduled" to a particular partition, it cannot be live-migrated, but needs to be deleted and re-created.

Alternative Solutions

An alternative to this is allowing providers access to the full list of objects, which might be okay with api-syncagent installations (they can do a label selector on their side), but would likely not be sufficient in a multi-provider marketplace situation.

Want to contribute?

• [ ] I would like to work on this issue.

Additional Context

No response

[embik]

This is waiting for #3494 to make sure we don't create conflicts.

[xmudrii]

I'm interested to work on this /assign

[sttts]

There is a partition API that can create slices https://github.com/kcp-dev/kcp/blob/main/staging/src/github.com/kcp-dev/sdk/apis/topology/v1alpha1/types_partition.go. Where is the connection to this? Having two partitioning concepts is confusing if they don't work together somehow.

[sttts]

The more I think about it the more I dislike it. An endpoint slice is not about VWs, it just lists VW URLs. So defining the partition there is strange.

Also, the topology API was intentionally built as a way to decouple slicing/partitioning from the endpoint slice logic. The later is just an output.

Last but not least, it's unclear to me how a label selector maps to a VW URL.

If one would want to build this, maybe a LabelSelectorPartition in the topology group could make sense, with a special APIExportLabelEndpointSlice VW. But mixing this into APIExportEndpointSlice feels wrong.

[sttts]

Or in other words: today nothing around topology and endpoint slices influences the existence of VW endpoints. The endpoint is there from the beginning just because the APIExport exists.

Plugging in a label mechanism is strange.

I believe we have built the whole topology machinery in a generic way that can handle other kind of partitions. One could build APIExportLabelPartition(Set) and then have APIExportLabelEndpointSlices with their own VW. This could be built outside or inside of kcp.

The existence of APIExportLabelPartition would have to be the trigger to serve a VW with the defined labels.

[embik]

I think that's a reasonable take, I was just sketching out how this could work in my head. A set of dedicated resources might be desirable, I just wonder if it makes sense given its essentially the same virtual workspace code, just with an additional label selector. But we can probably generalize the module used for this.

@xmudrii was planning to pick this up, so curious to hear his opinion

And the more I think about it, the naming should indeed avoid partitions. It will be confusing to talk about.