[Feature] FederatedResourceQuota Enhancement

[mszacillo]

Summary This feature aims to enhance the existing FederatedResourceQuota so it can impose namespaced resource limits directly on the Karmada control-plane level. This feature enhancement has several benefits, including:

1. Track resource consumption and limits from a unified place
2. Prevent users from over-consuming aggregate resources across clusters
3. Easier for users of application and cluster failover to ensure that fallback clusters have enough resources to house all the user's workflows

Proposal:

• [X] FederatedResourceQuota Enhancement: #6181 (@mszacillo)

Iteration Tasks -- Part-1: FederatedResourceQuota Enhancement MVP

• [X] Feature Flag addition: FederatedQuotaEnforcement #6366 (@liwang0513)
• [X] [API Change] Clarify intent and usage of staticAssignments in the FRQ API #6351 (@RainbowMango)
• [X] [Controller change] Adjust the behavior of the legacy staticAssignments to skip creating redundant ResourceQuotas to member clusters #6363 (@zhzhuang-zju)
• [X] Update federated-resource-quota-status-controller so that staticAssignment only updates AggregatedStatus #6382 (@liwang0513)
• [X] Create federated-resource-quota-enforcement controller: #6367 (@mszacillo) - Watch create/update events for FRQ, and update federatedresourcequota.status.overall - Watch deletion of RB -> find related FRQ -> recalculate quota usage - Periodically recalculate FRQ status
• [X] Add ResourceBinding validation webhook to monitor creates + updates to resourcebinding.spec and resourcebinding.spec.replicaRequirements after scheduling result #6377 (@seanlaii) - Check delta in quota usage and deny request if above limits - Otherwise update federatedresourcequota.status.overallUsed and then approve update request
• [X] Periodically reconcile resource quota enforcement controller: #6407 (@zhzhuang-zju)
• [X] Validate FederatedResourceQuota length to no more than 63 chars (#2168)
• [X] Add new feature documentation #835 (@mszacillo)
• [X] Functional verification and add e2e tests #6408 (@zhzhuang-zju)

Iteration Tasks -- Part-2: Further Enhancements beyond MVP

• [ ] [API Change] Add Scope to the FederatedResourceQuotaSpec
• [ ] Consider improvements to reconcile strategy for overall FRQ status controller (use cache instead of listing all RB each reconcile)
• [ ] Add quota exceeded events and conditions in case detector or scheduler fail to update the ResourceBinding (@mszacillo)
• [ ] FederatedResourceQuota controller should requeue unschedulable RBs to be scheduled in the case of a change to overall quota usage
• [ ] Determine if changes necessary to scheduler when reacting to denied RB updates
• [ ] Extract common resources usage calculation function to binding utils (@seanlaii)

Iteration Tasks -- Additional Follow-ups

• [ ] Allow enabling feature gates of karmada-webhook in karmadactl (@seanlaii)
• [X] Allow enabling feature gates of karmada-webhook in chart #6384 (@seanlaii)
• [X] Allow enabling feature gates of karmada-webhook in karmada-operator #6385 (@seanlaii)
• [ ] Consider supporting ResourceLimit in ReplicaRequirements

[mszacillo]

@RainbowMango Let me know if this looks alright!

Also, @zhzhuang-zju, do you have any preference on how to subdivide this work between us?

[RainbowMango]

Generally looks great! Thanks. I will leave comments if I want to twist or add something. For now, I'm thinking of introducing a feature gate, like what we did for the StatefulFailoverInjection, do you have any suggestion for the feature name?

[RainbowMango]

@mszacillo The origin behavior of staticAssignments might be a distraction for this feature as with this field unset, the controller still creates an empty ResourceQuota to all clusters, that doesn't make sense, because empty ResourceQuota can not do anything.

Sent #6351 for updating the comments, please take a look, and proposing two iterated tasks:

• (API) Adjust the behavior of the legacy staticAssignments to skip creating redundant ResourceQuotas to member clusters.
• (Controller) Adjust the behavior of the legacy staticAssignments to skip creating redundant ResourceQuotas to member clusters.

[mszacillo]

Great! I've gone ahead and updated the iteration tasks to include the two other staticAssignments tasks.

Just to double check, do these iteration tasks make sense as well:

• Update federated-resource-quota-status-controller to skip reconcile if staticAssignment is not set
• FederatedResourceQuota validating webhook should block updates to enable staticAssignment on Update requests

The thought it is that the staticAssignment controller and new federated resource quota controller will both attempt to update the status of the federated resource quota resource. So we want to prevent users from toggling between the two features. If someone wants to use staticAssignment, they can delete their quota and reapply using staticAssignment.

For now, I'm thinking of introducing a feature gate, like what we did for the StatefulFailoverInjection, do you have any suggestion for the feature name?

Perhaps FederatedQuotaEnforcement? Or GlobalQuotaEnforcement?

[RainbowMango]

The thought it is that the staticAssignment controller and new federated resource quota controller will both attempt to update the status of the federated resource quota resource.

This is a technical issue and can be resolved. Concurrent updates to the same resource object by multiple controllers are a common scenario. By clearly defining the responsibilities of each controller to ensure they do not modify the same fields simultaneously, we can resolve this problem.

In this case, we can let the federated-resource-quota-status-controller only update the .status.aggregatedStatus: https://github.com/karmada-io/karmada/blob/32a40764ea75b0f2ec1da0b6e6929195e07e0000/pkg/apis/policy/v1alpha1/federatedresourcequota_types.go#L103-L105, and the .status.overallUsed will be managed by the new controller.

FederatedResourceQuota validating webhook should block updates to enable staticAssignment on Update requests

So, we might not need to do this. The staticAssignments field is still an optional field. Once it is set, the origin controller will create ResourceQuota for member clusters.

Update federated-resource-quota-status-controller to skip reconcile if staticAssignment is not set

Basically yes, both the federated-resource-quota-status-controller and federated-resource-quota-sync-controller are for reconciling staticAssignment (I'm thinking of renaming it, by the way).

But there is a corner case that we probably can not just skip reconciliation even staticAssignment is unset, consider the following case: what if the staticAssignment is removed during an update operation, if we skip the reconciliation, the status(.status.aggregatedStatus) will not be cleaned up.

[RainbowMango]

Perhaps FederatedQuotaEnforcement? Or GlobalQuotaEnforcement?

I like the FederatedQuotaEnforcement as the feature name and feature gate name.

[mszacillo]

In this case, we can let the federated-resource-quota-status-controller only update the .status.aggregatedStatus:

Gotcha, I should have been more clear. The concern wasn't that two controllers are editing the status, but rather that they will be editing the same field OverallUsed. I'll edit the iteration tasks to account for this!

So, we might not need to do this. The staticAssignments field is still an optional field. Once it is set, the origin controller will create ResourceQuota for member clusters.

But there is a corner case that we probably can not just skip reconciliation even staticAssignment is unset, consider the following case: what if the staticAssignment is removed during an update operation, if we skip the reconciliation, the status(.status.aggregatedStatus) will not be cleaned up.

Makes sense! I'll edit this iteration task as well. Should I include the renaming of staticAssignment as part of this umbrella list, or should it be standalone?

[RainbowMango]

I'll edit this iteration task as well. Should I include the renaming of staticAssignment as part of this umbrella list, or should it be standalone?

~~OK, we can have one.~~ Refactoring/Twisting the legacy feature is also part of the new feature.

[edit]: Renaming the legacy controller may not be mandatory. Given that the new controller might be named federated-resource-quota-enforcement-controller, it seems unlikely to cause much misunderstanding. Then we will have 3 controllers:

• (legacy) federated-resource-quota-sync-controller: synchronizes Kubernetes ResourceQuota to member clusters based on staticAssignment field
• (legacy) federated-resource-quota-status-controller: aggregates and reports the usage status of ResourceQuota across all member clusters.
• (new) federated-resource-quota-enforcement-controller: Calculates and reports the usage based on the enforcement at the Federation side.

What do you think? Do you have any preference for the new controller name? @mszacillo @zhzhuang-zju

[RainbowMango]

By the way, please assign the task to yourself and @liwang0513 once you have selected it. PS: Tasks without an assignment are open for anyone to pick.

[zhzhuang-zju]

@mszacillo Thanks for your work. The tasks you have outlined are very specific and detailed.

Based on that, I only have a few minor comments:

• Add feature documentation and testing tasks to Iteration Tasks -- Part-1.
• User awareness: If the detector fails to update the ResourceBinding, or if the karmada-scheduler fails to update the ResourceBinding, we can improve user awareness by using events and conditions. This can also be added to Iteration Tasks -- Part-1.
• Can the task FederatedResourceQuota controller should requeue unschedulable RBs to be scheduled in the case of a change to overall quota usage be moved to Iteration Tasks -- Part-2? Preventing the scheduler from repeatedly attempting to schedule RBs that would exceed the remaining quota, and figuring out how to requeue unschedulable RBs, is more of a performance optimization for this feature. The design and development effort for this part is relatively significant, so it could be excluded from Part-1 and handled simply with retry-on-failure logic for now.

Besides, should we limit the resource names that FederatedResourceQuota supports? FYI, there are three types of ResourceQuota: Compute Resource Quota, Storage Resource Quota, and Object Count Quota. In Iteration Tasks -- Part-1, we may only be able to support Compute Resource Quota.

Also, @zhzhuang-zju, do you have any preference on how to subdivide this work between us?

Please feel free to pick the tasks you're interested in. I'm glad to join in design discussions and help with code reviews. If I see something I like, I’ll just assign it to myself too.

[zhzhuang-zju]

@mszacillo @RainbowMango I would like to improve the output of kubectl/karmadactl get federatedResourceQuota. The current output looks like this:

✗ kubectl get federatedresourcequotas.policy.karmada.io                                       
NAME   AGE
test   3m59s

There is no useful information currently. We can also add the overall and overallUsed fields to the output, like:

kubectl get federatedresourcequotas.policy.karmada.io
NAME   OVERALL          OVERALL_USED
test   {"cpu":"100m"}   

If that makes sense to you, we can include this task in Iteration Tasks -- Part-1.

[zhzhuang-zju]

[Controller change] Adjust the behavior of the legacy staticAssignments to skip creating redundant ResourceQuotas to member clusters.

@mszacillo Can you assign this task to me? Thanks~

[mszacillo]

Sounds great. I have updated the iteration tasks! Will start preparing necessary PRs shortly.

[RainbowMango]

Proposal one more taskValidate FederatedResourceQuota length to no more than 63 chars (#2168), @likakuli sent about 3 years ago :).

[seanlaii]

Hi, I am interested in

Add ResourceBinding validation webhook to monitor updates to resourcebinding.spec and resourcebinding.spec.replicaRequirements after scheduling result

• Check delta in quota usage and deny request if above limits
• Otherwise update federatedresourcequota.status.overallUsed and then approve update request

, could I take it?

[mszacillo]

@seanlaii Please go ahead! I will assign it to you. Let me know if you have questions - we can discuss more on Monday. :)

[RainbowMango]

Add quota exceeded events and conditions in case detector or scheduler fail to update the ResourceBinding

Shall we move this task to the next iteration? We might need to do some tests against the MVP version to figure out the potential effect.

[zhzhuang-zju]

Shall we move this task to the next iteration? We might need to do some tests against the MVP version to figure out the potential effect.

I'm ok with this

[mszacillo]

@RainbowMango @zhzhuang-zju I moved the iteration task to part 2! I'll start working on some documentation for this feature.

[zhzhuang-zju]

I'll start working on some documentation for this feature.

Great! Looking forward to it. I can help with translating the docs once they're ready.

[seanlaii]

Hi @mszacillo , could you help add the following tasks to the iteration as well? You could assign them to me. Thanks!

1. Enable feature gates options:

• Allow enabling feature gates of karmada-webhook in karmadactl.
• Allow enabling feature gates of karmada-webhook in chart.
• Allow enabling feature gates of karmada-webhook in karmada-operator. References: https://github.com/karmada-io/karmada/pull/6377#discussion_r2090667417

2. Extract common resources usage calculation function to binding utils. References: https://github.com/karmada-io/karmada/pull/6377#discussion_r2091028762 https://github.com/karmada-io/karmada/pull/6377#issuecomment-2883059490

[zhzhuang-zju]

With everyone's efforts, the basic functionality of the FederatedResourceQuota enhancement has been completed :tada: There are still two weeks until the v1.14 release, during which we can focus on strengthening and testing the existing features.

[zhzhuang-zju]

@mszacillo I can take on the task of functional verification and add e2e tests for the FRQ.

[mszacillo]

@zhzhuang-zju Of course, updated the umbrella ticket!

[RainbowMango]

Hi @mszacillo, shall we open another issue to track the tasks from part 2? I'm asking because this issue is tagged with v1.14, and in the release-1.14 backlog, each issue cannot have two tags and cannot exist in two backlogs.

[mszacillo]

@RainbowMango Sounds good! I'll close this issue and reopen a new one with the phase 2 tasks.

[RainbowMango]

Hi @CaesarTY, This feature has been included in the v1.14.0, I guess it is also applicable to your scenario, as we talked at KubeCon London 2025. Are you able to run a test with it? Any feedback would be appreciated, thanks!

[CaesarTY]

Hey @RainbowMango, we will onboard to Karmada this quarter with federatedResourceQuota feature, will get back to you once we have setup everything and test it. Thank you so much for the effort on this amazing feature!

[RainbowMango]

Awesome! Just take your time! @mszacillo @seanlaii @zhzhuang-zju deserve the credit here, I just participated in some discussion and review.