[FEATURE] Restrict Load balancer IP usage in the guest clusters

[jtamas96]

Currently if there is a DHCP on the network, handing out load balancer IPs cannot be restricted in the guest kubernetes clusters, so users can use up too many IPs on the cluster with the DHCP solution.

It would be good to have a feature to restrict the number of IPs for namespaces in the guest clusters, or even disable DHCP IP requests as a whole.

[w13915984028]

@jtamas96 Do you have more details about your Harvester cluster version, the steps to produce/reproduce, the related yaml output the CRD objects ... ? thanks.

[starbops]

Though the feature is not well-integrated into Harvester, there are manual ways to restrict the number of IP addresses being requested. It depends on where you want to enforce such a restriction. The main idea is to add a limitation for the LoadBalancer type of Service or LoadBalancer objects that could be created. This is done by specifying the corresponding values in the ResourceQuota object for the namespace in which users will create LB-type services or the Project containing the Kubernetes guest cluster.

1. Adding the restriction against the LoadBalancer type of Service objects that could be created on the Kubernetes guest cluster. By specifying the number of LB type of Service objects allowed to be created, we can therefore limit the number of IP addresses being requested via DHCP. Such a restriction only applies to the namespace the ResourceQuota object is associated with. For example, the following test-rq ResourceQuota restricts the number of LB-type Services that can be created to one for the test-ns namespace:

   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: test-rq
     namespace: test-ns
   spec:
     hard:
       services.loadbalancers: "1" # <-- means the maximum allowed number of LB-type of Services is 1
   

2. Adding the restriction against the LoadBalancer objects that could be created on the Harvester cluster. This allows users to restrict the number of LoadBalancer objects being created on the Harvester cluster for a specific namespace. When a Kubernetes guest cluster is created under that specific namespace, the number of LB-type of Service objects that could be created across all the namespaces on the guest cluster will be restricted by the ResourceQuota object. This is the only namespace-agnostic way to restrict the number of LB-type of Service objects being created because Harvester creates LoadBalancer objects for each LB-type of Service object created on the guest cluster under the hood. For example, the following default-csx6j ResourceQuota restricts the number of LoadBalancer objects that can be created to one for the test-ns namespace:

   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: default-csx6j
     namespace: test-ns
   spec:
     hard:
       count/loadbalancers.loadbalancer.harvesterhci.io: "1" # <-- means the maximum allowed number of LBs is 1
   

[jtamas96]

@jtamas96 Do you have more details about your Harvester cluster version, the steps to produce/reproduce, the related yaml output the CRD objects ... ? thanks.

My version is 1.3.0 If you just create a LoadBalancer service in Rancher, from a guest kubernetes cluster, and you have a dhcp on the network you can reproduce this. Every time a LoadBalancer is created, it will get an IP from the DHCP.

Though the feature is not well-integrated into Harvester, there are manual ways to restrict the number of IP addresses being requested. It depends on where you want to enforce such a restriction. The main idea is to add a limitation for the LoadBalancer type of Service or LoadBalancer objects that could be created. This is done by specifying the corresponding values in the ResourceQuota object for the namespace in which users will create LB-type services or the Project containing the Kubernetes guest cluster.

1. Adding the restriction against the LoadBalancer type of Service objects that could be created on the Kubernetes guest cluster. By specifying the number of LB type of Service objects allowed to be created, we can therefore limit the number of IP addresses being requested via DHCP. Such a restriction only applies to the namespace the ResourceQuota object is associated with. For example, the following test-rq ResourceQuota restricts the number of LB-type Services that can be created to one for the test-ns namespace:

   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: test-rq
     namespace: test-ns
   spec:
     hard:
       services.loadbalancers: "1" # <-- means the maximum allowed number of LB-type of Services is 1
   

2. Adding the restriction against the LoadBalancer objects that could be created on the Harvester cluster. This allows users to restrict the number of LoadBalancer objects being created on the Harvester cluster for a specific namespace. When a Kubernetes guest cluster is created under that specific namespace, the number of LB-type of Service objects that could be created across all the namespaces on the guest cluster will be restricted by the ResourceQuota object. This is the only namespace-agnostic way to restrict the number of LB-type of Service objects being created because Harvester creates LoadBalancer objects for each LB-type of Service object created on the guest cluster under the hood. For example, the following default-csx6j ResourceQuota restricts the number of LoadBalancer objects that can be created to one for the test-ns namespace:

   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: default-csx6j
     namespace: test-ns
   spec:
     hard:
       count/loadbalancers.loadbalancer.harvesterhci.io: "1" # <-- means the maximum allowed number of LBs is 1
   

Thanks for the hint! I think it would be a good feature to integrate something like this into Harvester itself.

[ibrokethecloud]

@starbops @bk201 should we look at a rancher UI enhancement to allow this at the project level?

[bk201]

@ibrokethecloud ~~the feature should be implemented in 1.3.0: https://github.com/harvester/harvester/issues/4981 load balancer is configurable~~:

Update: Ignore my previous statement. Those settings are for the host cluster and it only supports configuring services.loadbalancers

[w13915984028]

@jtamas96

If you just create a LoadBalancer service in Rancher, from a guest kubernetes cluster, and you have a dhcp on the network you can reproduce this. Every time a LoadBalancer is created, it will get an IP from the DHCP.

I guess your idea is to set sort of limitation on Rancher side, limit the resource of guest cluster. Thus a user can't occupy too much resurces.

This looks to be a general requirements when creating guest cluster from Rancher: e.g. Node/VM count , Node/VM CPU/Memory/Disk ..., LB count ...

It needs to cooperation between Rancher and Harvester / other node driver.

We will take a further look, thanks.

[w13915984028]

@jtamas96 Do you have an detailed example, where the guest cluster is allocted to a speific user/customer, and what can the user operate in this guest cluster?

If this guest cluster is fully handed over to a user/customer, then adding resources quota in guest cluster seems not making sense.

[jtamas96]

Hi,

We are providing only namespaces for users, they don't have full access to the cluster.