capsule
capsule copied to clipboard
Published
20 hours ago •
projectcapsule
projectcapsule
Tenant Resource Quota admission controller doesn't block resources creation
Bug description
Tenant Resource Quota admission controller doesn't block resources creation
How to reproduce
As cluster admin:
cmp~/Shared/capsule$ kubectl get tenants
NAME NAMESPACE QUOTA NAMESPACE COUNT OWNER AGE
oil 3 3 alice 161m
A tenant owner:
$ kubectl -n oil-production get quota
NAME AGE REQUEST LIMIT
capsule-oil-0 161m requests.cpu: 0/8, requests.memory: 0/16Gi limits.cpu: 0/8, limits.memory: 0/16Gi
capsule-oil-1 161m pods: 0/10
capsule-oil-2 161m requests.storage: 0/100Gi
$ kubectl -n oil-development get quota
NAME AGE REQUEST LIMIT
capsule-oil-0 161m requests.cpu: 0/8, requests.memory: 0/16Gi limits.cpu: 0/8, limits.memory: 0/16Gi
capsule-oil-1 161m pods: 0/10
capsule-oil-2 161m requests.storage: 0/100Gi
$ kubectl -n oil-test get quota
NAME AGE REQUEST LIMIT
capsule-oil-0 161m requests.cpu: 0/8, requests.memory: 0/16Gi limits.cpu: 0/8, limits.memory: 0/16Gi
capsule-oil-1 161m pods: 0/10
capsule-oil-2 161m requests.storage: 0/100Gi
check used quota at tenant level
$ kubectl -n oil-production get quota capsule-oil-1 -o yaml | grep quota.capsule.clastix.io/used-pods
quota.capsule.clastix.io/used-pods: "0"
scale up pods
kubectl -n oil-production scale rs nginx --replicas=3
kubectl -n oil-development scale rs nginx --replicas=3
kubectl -n oil-test scale rs nginx --replicas=5
and check quota:
kubectl -n oil-production get quota capsule-oil-1 -o yaml | grep quota.capsule.clastix.io/used-pods
quota.capsule.clastix.io/used-pods: "11"
kubectl -n oil-production describe quota capsule-oil-1
Name: capsule-oil-1
Namespace: oil-production
Resource Used Hard
-------- ---- ----
pods 3 3 <-- see hard quota!!!
kubectl -n oil-test describe quota capsule-oil-1
Name: capsule-oil-1
Namespace: oil-test
Resource Used Hard
-------- ---- ----
pods 5 5
Expected behavior
Tenant Resource Quota admission controller should block resource creation crossing the set quota.
Logs
Capsule controller has been started with following args but I do see only INFO
containers:
- args:
- --metrics-addr=127.0.0.1:8080
- --enable-leader-election
- --zap-log-level=10
- --zap-devel
command:
- /manager
image: quay.io/clastix/capsule:latest
2020-08-10T10:22:00.793Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:22:00.942Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:22:00.942Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:22:00.942Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:22:01.093Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:22:01.093Z INFO controllers.Tenant Ensuring RoleBinding for owner {"Request.Name": "oil"}
2020-08-10T10:22:01.096Z INFO controllers.Tenant Role Binding sync result: updated {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:22:01.096Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:22:01.097Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:22:01.097Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:22:01.097Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:22:01.097Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:22:01.097Z INFO controllers.Tenant Ensuring Namespace count {"Request.Name": "oil"}
2020-08-10T10:22:01.107Z INFO controllers.Tenant Tenant reconciling completed {"Request.Name": "oil"}
2020-08-10T10:32:29.193Z INFO controllers.Tenant Starting processing of Namespaces {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:29.203Z INFO controllers.Tenant Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:29.204Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:29.220Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:29.220Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:29.252Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:29.252Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:29.262Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:29.262Z INFO controllers.Tenant Starting processing of Node Selector {"Request.Name": "oil"}
2020-08-10T10:32:29.263Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:32:29.263Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:32:29.263Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:32:29.263Z INFO controllers.Tenant Starting processing of Limit Ranges {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:29.263Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:29.290Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:29.290Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:29.330Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:29.330Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:29.340Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:29.341Z INFO controllers.Tenant Starting processing of Resource Quotas {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:29.342Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:29.358Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:29.358Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 30Mi {"Request.Name": "oil"}
2020-08-10T10:32:29.380Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:29.381Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:29.459Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:29.459Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 300Mi {"Request.Name": "oil"}
2020-08-10T10:32:29.485Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:29.485Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 300m {"Request.Name": "oil"}
2020-08-10T10:32:29.519Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:29.520Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:29.520Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 3 {"Request.Name": "oil"}
2020-08-10T10:32:29.546Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:32:29.546Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:29.546Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:29.565Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:32:29.565Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:29.568Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:29.568Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 30Mi {"Request.Name": "oil"}
2020-08-10T10:32:29.579Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:29.579Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:29.595Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:29.595Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 300Mi {"Request.Name": "oil"}
2020-08-10T10:32:29.607Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:29.607Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 300m {"Request.Name": "oil"}
2020-08-10T10:32:29.631Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:29.631Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:29.631Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 3 {"Request.Name": "oil"}
2020-08-10T10:32:29.748Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:32:29.748Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:29.748Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:29.919Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:32:29.919Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:29.950Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:29.950Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 30Mi {"Request.Name": "oil"}
2020-08-10T10:32:30.098Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:30.098Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:30.248Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:30.248Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 300Mi {"Request.Name": "oil"}
2020-08-10T10:32:30.398Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:30.398Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 300m {"Request.Name": "oil"}
2020-08-10T10:32:30.548Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:30.548Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:30.548Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 3 {"Request.Name": "oil"}
2020-08-10T10:32:30.697Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:32:30.697Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:30.697Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:30.850Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:32:30.850Z INFO controllers.Tenant Ensuring RoleBinding for owner {"Request.Name": "oil"}
2020-08-10T10:32:30.856Z INFO controllers.Tenant Role Binding sync result: updated {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:32:30.856Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:32:30.857Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:32:30.857Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:32:30.857Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:32:30.857Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:32:30.857Z INFO controllers.Tenant Ensuring Namespace count {"Request.Name": "oil"}
2020-08-10T10:32:30.871Z INFO controllers.Tenant Tenant reconciling completed {"Request.Name": "oil"}
2020-08-10T10:32:30.871Z INFO controllers.Tenant Starting processing of Namespaces {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:30.884Z INFO controllers.Tenant Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:30.884Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:30.887Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:30.887Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:30.891Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:30.891Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:30.894Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:30.894Z INFO controllers.Tenant Starting processing of Node Selector {"Request.Name": "oil"}
2020-08-10T10:32:30.894Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:32:30.895Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:32:30.895Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:32:30.895Z INFO controllers.Tenant Starting processing of Limit Ranges {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:30.895Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:30.898Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:30.898Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:30.903Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:30.903Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:30.907Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:30.907Z INFO controllers.Tenant Starting processing of Resource Quotas {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:30.907Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:30.912Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:30.912Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 300Mi {"Request.Name": "oil"}
2020-08-10T10:32:31.050Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:31.050Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 300m {"Request.Name": "oil"}
2020-08-10T10:32:31.198Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:31.198Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 30Mi {"Request.Name": "oil"}
2020-08-10T10:32:31.347Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:31.347Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:31.497Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:31.498Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:31.498Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 3 {"Request.Name": "oil"}
2020-08-10T10:32:31.647Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:32:31.648Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:31.648Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:31.797Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:32:31.797Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:31.846Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:31.846Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 30Mi {"Request.Name": "oil"}
2020-08-10T10:32:31.997Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:31.997Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:32.147Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:32.147Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 300Mi {"Request.Name": "oil"}
2020-08-10T10:32:32.303Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:32.303Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 300m {"Request.Name": "oil"}
2020-08-10T10:32:32.450Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:32.451Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:32.451Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 3 {"Request.Name": "oil"}
2020-08-10T10:32:32.600Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:32:32.601Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:32.601Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:32.747Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:32:32.747Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:32.795Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:32.796Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 30Mi {"Request.Name": "oil"}
2020-08-10T10:32:32.947Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:32.947Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:33.097Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:33.097Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 300Mi {"Request.Name": "oil"}
2020-08-10T10:32:33.247Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:33.247Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 300m {"Request.Name": "oil"}
2020-08-10T10:32:33.398Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:33.398Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:33.398Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 3 {"Request.Name": "oil"}
2020-08-10T10:32:33.546Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:32:33.547Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:33.547Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:33.697Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:32:33.697Z INFO controllers.Tenant Ensuring RoleBinding for owner {"Request.Name": "oil"}
2020-08-10T10:32:33.701Z INFO controllers.Tenant Role Binding sync result: updated {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:32:33.701Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:32:33.702Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:32:33.702Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:32:33.702Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:32:33.702Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:32:33.702Z INFO controllers.Tenant Ensuring Namespace count {"Request.Name": "oil"}
2020-08-10T10:32:33.712Z INFO controllers.Tenant Tenant reconciling completed {"Request.Name": "oil"}
2020-08-10T10:32:43.303Z INFO controllers.Tenant Starting processing of Namespaces {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:43.357Z INFO controllers.Tenant Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:43.358Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:43.388Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:43.388Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:43.410Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:43.410Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:43.427Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:43.427Z INFO controllers.Tenant Starting processing of Node Selector {"Request.Name": "oil"}
2020-08-10T10:32:43.427Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:32:43.428Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:32:43.428Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:32:43.428Z INFO controllers.Tenant Starting processing of Limit Ranges {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:43.428Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:43.441Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:43.441Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:43.447Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:43.447Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:43.460Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:43.460Z INFO controllers.Tenant Starting processing of Resource Quotas {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:43.460Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:43.463Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:43.463Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:43.499Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:43.499Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 600Mi {"Request.Name": "oil"}
2020-08-10T10:32:43.513Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:43.513Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:43.534Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:43.535Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 60Mi {"Request.Name": "oil"}
2020-08-10T10:32:43.547Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:43.547Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:43.547Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 6 {"Request.Name": "oil"}
2020-08-10T10:32:43.579Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:32:43.579Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:43.579Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:43.603Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:32:43.603Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:43.609Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:43.609Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:43.621Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:43.621Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 60Mi {"Request.Name": "oil"}
2020-08-10T10:32:43.632Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:43.632Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:43.644Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:43.644Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 600Mi {"Request.Name": "oil"}
2020-08-10T10:32:43.657Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:43.657Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:43.657Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 6 {"Request.Name": "oil"}
2020-08-10T10:32:43.717Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:32:43.717Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:43.717Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:43.898Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:32:43.899Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:43.914Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:43.914Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:44.066Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:44.066Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 600Mi {"Request.Name": "oil"}
2020-08-10T10:32:44.219Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:44.219Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:44.366Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:44.366Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 60Mi {"Request.Name": "oil"}
2020-08-10T10:32:44.517Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:44.517Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:44.517Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 6 {"Request.Name": "oil"}
2020-08-10T10:32:44.667Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:32:44.667Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:44.667Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:44.816Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:32:44.816Z INFO controllers.Tenant Ensuring RoleBinding for owner {"Request.Name": "oil"}
2020-08-10T10:32:44.820Z INFO controllers.Tenant Role Binding sync result: updated {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:32:44.820Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:32:44.821Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:32:44.821Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:32:44.821Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:32:44.821Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:32:44.821Z INFO controllers.Tenant Ensuring Namespace count {"Request.Name": "oil"}
2020-08-10T10:32:44.833Z INFO controllers.Tenant Tenant reconciling completed {"Request.Name": "oil"}
2020-08-10T10:32:44.833Z INFO controllers.Tenant Starting processing of Namespaces {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:44.845Z INFO controllers.Tenant Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:44.845Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:44.850Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:44.850Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:44.854Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:44.854Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:44.858Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:44.858Z INFO controllers.Tenant Starting processing of Node Selector {"Request.Name": "oil"}
2020-08-10T10:32:44.859Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:32:44.859Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:32:44.859Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:32:44.859Z INFO controllers.Tenant Starting processing of Limit Ranges {"Request.Name": "oil", "items": 1}
2020-08-10T10:32:44.859Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:44.863Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:44.863Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:44.866Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:44.866Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:32:44.871Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:44.871Z INFO controllers.Tenant Starting processing of Resource Quotas {"Request.Name": "oil", "items": 3}
2020-08-10T10:32:44.871Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:44.876Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:44.876Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:45.016Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:45.016Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 600Mi {"Request.Name": "oil"}
2020-08-10T10:32:45.166Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:45.166Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:45.315Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:45.315Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 60Mi {"Request.Name": "oil"}
2020-08-10T10:32:45.466Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:32:45.466Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:45.466Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 6 {"Request.Name": "oil"}
2020-08-10T10:32:45.615Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:32:45.615Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:45.615Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:45.765Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:32:45.765Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:45.814Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:45.814Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:45.965Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:45.965Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 60Mi {"Request.Name": "oil"}
2020-08-10T10:32:46.115Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:46.115Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:46.265Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:46.265Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 600Mi {"Request.Name": "oil"}
2020-08-10T10:32:46.416Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:32:46.416Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:46.416Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 6 {"Request.Name": "oil"}
2020-08-10T10:32:46.564Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:32:46.565Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:46.565Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:46.726Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:32:46.726Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:32:46.764Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:46.764Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 600m {"Request.Name": "oil"}
2020-08-10T10:32:46.918Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:46.918Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 60Mi {"Request.Name": "oil"}
2020-08-10T10:32:47.065Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:32:47.065Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 1200m {"Request.Name": "oil"}
2020-08-10T10:32:47.215Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:32:47.215Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 600Mi {"Request.Name": "oil"}
2020-08-10T10:32:47.366Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:32:47.366Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:32:47.366Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 6 {"Request.Name": "oil"}
2020-08-10T10:32:47.515Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:32:47.516Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:32:47.516Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:32:47.667Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:32:47.667Z INFO controllers.Tenant Ensuring RoleBinding for owner {"Request.Name": "oil"}
2020-08-10T10:32:47.673Z INFO controllers.Tenant Role Binding sync result: updated {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:32:47.674Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:32:47.674Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:32:47.674Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:32:47.674Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:32:47.674Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:32:47.674Z INFO controllers.Tenant Ensuring Namespace count {"Request.Name": "oil"}
2020-08-10T10:32:47.683Z INFO controllers.Tenant Tenant reconciling completed {"Request.Name": "oil"}
2020-08-10T10:33:11.273Z INFO controllers.Tenant Starting processing of Namespaces {"Request.Name": "oil", "items": 3}
2020-08-10T10:33:11.283Z INFO controllers.Tenant Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:33:11.284Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:11.292Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:11.292Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:11.318Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:11.318Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:11.347Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:11.347Z INFO controllers.Tenant Starting processing of Node Selector {"Request.Name": "oil"}
2020-08-10T10:33:11.347Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:33:11.347Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:33:11.347Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:33:11.347Z INFO controllers.Tenant Starting processing of Limit Ranges {"Request.Name": "oil", "items": 1}
2020-08-10T10:33:11.347Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:11.353Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:11.353Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:11.361Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:11.362Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:11.375Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:11.375Z INFO controllers.Tenant Starting processing of Resource Quotas {"Request.Name": "oil", "items": 3}
2020-08-10T10:33:11.375Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:33:11.414Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:11.414Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 1800m {"Request.Name": "oil"}
2020-08-10T10:33:11.507Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:11.507Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 900Mi {"Request.Name": "oil"}
2020-08-10T10:33:11.608Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:11.609Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 900m {"Request.Name": "oil"}
2020-08-10T10:33:11.654Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:11.654Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 90Mi {"Request.Name": "oil"}
2020-08-10T10:33:11.730Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:11.730Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:33:11.730Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 11 {"Request.Name": "oil"}
2020-08-10T10:33:11.769Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:33:11.769Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:33:11.769Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:33:11.806Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:33:11.806Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:33:11.822Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:11.822Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 2200m {"Request.Name": "oil"}
2020-08-10T10:33:11.869Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:11.869Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 1100Mi {"Request.Name": "oil"}
2020-08-10T10:33:11.889Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:11.889Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 1100m {"Request.Name": "oil"}
2020-08-10T10:33:11.925Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:11.925Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 110Mi {"Request.Name": "oil"}
2020-08-10T10:33:11.939Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:11.939Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:33:11.939Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 11 {"Request.Name": "oil"}
2020-08-10T10:33:12.000Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:33:12.001Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:33:12.001Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:33:12.010Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:33:12.010Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:33:12.014Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:12.014Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 2200m {"Request.Name": "oil"}
2020-08-10T10:33:12.135Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:12.135Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 1100Mi {"Request.Name": "oil"}
2020-08-10T10:33:12.282Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:12.282Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 1100m {"Request.Name": "oil"}
2020-08-10T10:33:12.434Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:12.434Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 110Mi {"Request.Name": "oil"}
2020-08-10T10:33:12.599Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:12.599Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:33:12.599Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 11 {"Request.Name": "oil"}
2020-08-10T10:33:12.731Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:33:12.731Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:33:12.731Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:33:12.881Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:33:12.882Z INFO controllers.Tenant Ensuring RoleBinding for owner {"Request.Name": "oil"}
2020-08-10T10:33:12.886Z INFO controllers.Tenant Role Binding sync result: updated {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:33:12.886Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:33:12.886Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:33:12.887Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:33:12.887Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:33:12.887Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:33:12.887Z INFO controllers.Tenant Ensuring Namespace count {"Request.Name": "oil"}
2020-08-10T10:33:12.898Z INFO controllers.Tenant Tenant reconciling completed {"Request.Name": "oil"}
2020-08-10T10:33:12.898Z INFO controllers.Tenant Starting processing of Namespaces {"Request.Name": "oil", "items": 3}
2020-08-10T10:33:12.909Z INFO controllers.Tenant Starting processing of Network Policies {"Request.Name": "oil", "items": 1}
2020-08-10T10:33:12.909Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:12.914Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:12.914Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:12.917Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:12.917Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/network-policy notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:12.921Z INFO controllers.Tenant Network Policy sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:12.921Z INFO controllers.Tenant Starting processing of Node Selector {"Request.Name": "oil"}
2020-08-10T10:33:12.921Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-development"}
2020-08-10T10:33:12.921Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-production"}
2020-08-10T10:33:12.921Z INFO controllers.Tenant Namespace Node sync result: unchanged {"Request.Name": "oil", "name": "oil-test"}
2020-08-10T10:33:12.921Z INFO controllers.Tenant Starting processing of Limit Ranges {"Request.Name": "oil", "items": 1}
2020-08-10T10:33:12.921Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:12.925Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:12.925Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:12.928Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:12.928Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/limit-range notin (0) {"Request.Name": "oil"}
2020-08-10T10:33:12.931Z INFO controllers.Tenant LimitRange sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:12.931Z INFO controllers.Tenant Starting processing of Resource Quotas {"Request.Name": "oil", "items": 3}
2020-08-10T10:33:12.931Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:33:12.935Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:12.935Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 1100Mi {"Request.Name": "oil"}
2020-08-10T10:33:13.082Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:13.082Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 1100m {"Request.Name": "oil"}
2020-08-10T10:33:13.237Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:13.237Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 110Mi {"Request.Name": "oil"}
2020-08-10T10:33:13.399Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:13.399Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 2200m {"Request.Name": "oil"}
2020-08-10T10:33:13.531Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-development"}
2020-08-10T10:33:13.532Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:33:13.532Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 11 {"Request.Name": "oil"}
2020-08-10T10:33:13.680Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-development"}
2020-08-10T10:33:13.681Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:33:13.681Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:33:13.830Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-development"}
2020-08-10T10:33:13.830Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:33:13.880Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:13.880Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 1100Mi {"Request.Name": "oil"}
2020-08-10T10:33:14.032Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:14.032Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 1100m {"Request.Name": "oil"}
2020-08-10T10:33:14.181Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:14.181Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 110Mi {"Request.Name": "oil"}
2020-08-10T10:33:14.330Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:14.330Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 2200m {"Request.Name": "oil"}
2020-08-10T10:33:14.481Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-production"}
2020-08-10T10:33:14.482Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:33:14.482Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 11 {"Request.Name": "oil"}
2020-08-10T10:33:14.630Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-production"}
2020-08-10T10:33:14.630Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:33:14.630Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:33:14.781Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-production"}
2020-08-10T10:33:14.781Z INFO controllers.Tenant Pruning objects with label selector capsule.clastix.io/resource-quota notin (0,1,2) {"Request.Name": "oil"}
2020-08-10T10:33:14.829Z INFO controllers.Tenant Desired hard limits.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:14.829Z INFO controllers.Tenant Computed limits.cpu quota for the whole Tenant is 2200m {"Request.Name": "oil"}
2020-08-10T10:33:14.981Z INFO controllers.Tenant Desired hard limits.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:14.981Z INFO controllers.Tenant Computed limits.memory quota for the whole Tenant is 1100Mi {"Request.Name": "oil"}
2020-08-10T10:33:15.130Z INFO controllers.Tenant Desired hard requests.cpu quota is 8 {"Request.Name": "oil"}
2020-08-10T10:33:15.130Z INFO controllers.Tenant Computed requests.cpu quota for the whole Tenant is 1100m {"Request.Name": "oil"}
2020-08-10T10:33:15.280Z INFO controllers.Tenant Desired hard requests.memory quota is 16Gi {"Request.Name": "oil"}
2020-08-10T10:33:15.281Z INFO controllers.Tenant Computed requests.memory quota for the whole Tenant is 110Mi {"Request.Name": "oil"}
2020-08-10T10:33:15.431Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-0", "namespace": "oil-test"}
2020-08-10T10:33:15.431Z INFO controllers.Tenant Desired hard pods quota is 10 {"Request.Name": "oil"}
2020-08-10T10:33:15.431Z INFO controllers.Tenant Computed pods quota for the whole Tenant is 11 {"Request.Name": "oil"}
2020-08-10T10:33:15.580Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-1", "namespace": "oil-test"}
2020-08-10T10:33:15.581Z INFO controllers.Tenant Desired hard requests.storage quota is 100Gi {"Request.Name": "oil"}
2020-08-10T10:33:15.581Z INFO controllers.Tenant Computed requests.storage quota for the whole Tenant is 0 {"Request.Name": "oil"}
2020-08-10T10:33:15.730Z INFO controllers.Tenant Resource Quota sync result: unchanged {"Request.Name": "oil", "name": "capsule-oil-2", "namespace": "oil-test"}
2020-08-10T10:33:15.730Z INFO controllers.Tenant Ensuring RoleBinding for owner {"Request.Name": "oil"}
2020-08-10T10:33:15.734Z INFO controllers.Tenant Role Binding sync result: updated {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-development"}
2020-08-10T10:33:15.734Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-development"}
2020-08-10T10:33:15.734Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-production"}
2020-08-10T10:33:15.734Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-production"}
2020-08-10T10:33:15.734Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:admin", "namespace": "oil-test"}
2020-08-10T10:33:15.735Z INFO controllers.Tenant Role Binding sync result: unchanged {"Request.Name": "oil", "name": "namespace:deleter", "namespace": "oil-test"}
2020-08-10T10:33:15.735Z INFO controllers.Tenant Ensuring Namespace count {"Request.Name": "oil"}
2020-08-10T10:33:15.743Z INFO controllers.Tenant Tenant reconciling completed {"Request.Name": "oil"}
Additional context
- Capsule version: (
capsule --version) - Kubernetes version: (
kubectl version)
I'm aware of this behavior and it's a viable trade-off, although not entirely satisfying.
I'm still trying to figure out how to apply these constraints without reinventing the wheel and ensuring the hard quota is absolutely honored.
Right now, Capsule is setting the Tenant quotas to each Namespace handled by it and watching for events: once the sum of the used resources across all the Namespace resources is exceeded, we're blocking them as you noticed.
Using this poor trick works well since we don't need a further CRD, and no need to listen on multiple Validating webhooks since there could be tons of resources handled by Quota ones.
Regarding #14, I guess it's not related: as you said, the update is processed correctly.
Let's keep this one open and try to figure out in the future how to implement a better strategy.
Hi @prometherion, I think it's going to be important to address this problem in a future release. While it may be viable for use to scope a tenant quota to eg. hosts which kinda hard enforce the tenant quota, we should honour the resource stretched on a tenant. I am happy to develop this with you.
The last call was, that we are looking for new ideas how to solve, while I am certainly not experienced enough to come up with a smart idea, I asked ChatGPT what it thinks about the problem. Here are it's best thoughts:
Here are a few ideas for how you might address these issues:
Enforce quotas at the API level: Instead of relying on annotations to enforce quotas, you could use the Kubernetes API server to enforce quotas directly. The API server has built-in support for resource quotas, and you can use it to enforce limits on the number of objects that can be created in a namespace, as well as limits on the resource usage of those objects. By enforcing quotas at the API level, you can ensure that they can't be bypassed or modified by users.
Use resource quotas in combination with limit ranges: Limit ranges allow you to specify minimum and maximum resource limits for objects in a namespace. You can use limit ranges in combination with resource quotas to provide a more fine-grained level of control over resource allocation. For example, you could use a limit range to set a minimum limit for CPU and memory usage, and then use a resource quota to set a maximum limit. This way, you can ensure that each tenant has access to a minimum level of resources, while still enforcing a maximum limit to prevent overuse.
Implement a queueing system for resource allocation: If you are concerned about racing conditions when multiple tenants are trying to allocate resources simultaneously, you could implement a queueing system to manage resource allocation requests. This could be as simple as a first-in, first-out queue, or you could use a more sophisticated system that takes into account factors such as the priority of the request or the available resources in the cluster. By using a queueing system, you can ensure that resource allocation requests are processed in a predictable and fair manner.
idea 3 seems interesting. tell me more
Define a custom resource definition (CRD) for resource allocation requests. This resource should contain metadata about the tenant making the request, the resources being requested, and any other relevant information.
Create a controller for the resource allocation request CRD. This controller will watch for new resource allocation requests and add them to a queue.
Implement a resource allocation process that runs periodically (e.g., every minute). This process should dequeue resource allocation requests from the queue and process them one by one.
When processing a resource allocation request, the resource allocation process should check if the requested resources are available in the cluster. If they are not, the request should be rejected. If the resources are available, the process should create the requested objects (e.g., pods, services, etc.) and update the tenant's resource quota to reflect the additional usage.
To ensure that the queue is processed in a predictable and fair manner, you might want to consider implementing priority levels for resource allocation requests. For example, you could give higher priority to requests from tenants with lower usage or to requests that are requesting fewer resources. You could also implement a maximum queue size to prevent the queue from growing too large and consuming too many resources.
Do you think something similar is achievable without a dedicated controller? We tried to overcome this odd scenario by creating a validation webhook to stop the update of the ResourceQuota object, thinking this would have prevented the odd scenario
It is possible to implement a queueing system for resource allocation without using a dedicated controller, but it will likely be more complex and may not provide the same level of control and guarantees as a dedicated controller.One approach you could take is to use a combination of Kubernetes custom resources, admission webhooks, and a separate application to manage resource allocation requests. Here's an overview of how this might work:
Define a custom resource definition (CRD) for resource allocation requests. This resource should contain metadata about the tenant making the request, the resources being requested, and any other relevant information.
Implement an admission webhook that intercepts resource allocation requests and adds them to a queue. The admission webhook could be implemented as a separate application that communicates with the Kubernetes API server using the webhook admission control mechanism.
Implement a resource allocation process that runs periodically (e.g., every minute). This process should dequeue resource allocation requests from the queue and process them one by one.
When processing a resource allocation request, the resource allocation process should check if the requested resources are available in the cluster. If they are not, the request should be rejected. If the resources are available, the process should create the requested objects (e.g., pods, services, etc.) and update the tenant's resource quota to reflect the additional usage.
Using this approach, you can intercept resource allocation requests and add them to a queue before they are processed by the Kubernetes API server. This allows you to implement a more sophisticated resource allocation process that can take into account factors such as the priority of the request or the available resources in the cluster.
However, it's important to note that this approach may not provide the same level of control and guarantees as a dedicated controller. For example, if the resource allocation process fails or is unable to complete a request, it may be difficult to handle the error and ensure that the request is eventually processed. Additionally, the admission webhook may not be able to intercept all resource allocation requests, depending on the scope of the webhook and the types of objects being created.
Already some interesting approaches. I ll have some more conversations and see what comes around.. :)
All the proposals suggested by ChatGPT are valuable and not effective for a production use case since missing the knowledge of the underlying internals of Kubernetes.
The problem with the following issue is the fact that a ResourceQuota object is controlled by an admission controller (named ResourceQuota) and there's no way in preventing with a validation webhook the change of that object in order to calculate, in advance, if the resource request (like the creation of a pod, or a volume) is allowed or not.
The basic idea behind Capsule has been simplicity and reusing the same Kubernetes API primitives, trying to deliver new ones at minimum reducing the cognitive load in operating a multi-tenant environment.
There are just two concrete ways to address this:
- implementing yet another ResourceQuota admission controller
This is cumbersome and repetitive, also considering that there's already an implementation, although non-Tenant aware.
- creating a Kubernetes Enhancement Proposal supporting a GlobalResourceQuota
Essentially, it would be pretty similar to ResourceQuota but living at the cluster-scope level, with the same specification, but with a label selector that we could use to address the Tenant namespaces.
The latter one is the most reasonable solution to me, although requiring to deal with the community and finding its consensus, besides making the fix compatible only starting from the release where the new resource is going to land. If you'd like to help me in drafting a KEP eager to work together, as well as attend the community meeting.
Just read about SchedulingGates as new feature gate:
https://kubernetes.io/blog/2022/12/26/pod-scheduling-readiness-alpha/
This could be interesting. In the use case the mention this:
Scheduling gates allows an external quota manager to address the above limitation of ResourceQuota. Specifically, the manager could add a example.com/quota-check scheduling gate to all Pods created in the cluster (using a mutating webhook). The manager would then remove the gate when there is quota to start the Pod.
I will have to play around with this feature a bit, maybe delaying scheduling might already help. But i obv. would also love to create a KEP :3 !
Yeah, this could be interesting, although remember that we're using ResourceQuota for multiple resources like PVC, Secret, PV, ConfigMap and, in the end, Pods.