The kubernetes node CPU useage should subtract the CPU usage of services that use EXT resources

[shijieqin]

Describe the feature

1. Ext-resource service(The service using EXT resources) is to populate the idle resources of the kubernetes node. If the CPU used by the ext-resource service is calculated to the CPU of the kubernetes node, nodeResourceController will double-compute the CPU used of the ext-resource service when updating the kubernetes node ext resources (the ext-resource of the service requested has been calculated into the allocation by the kubelet)
2. Crane-Agent should expose the CPU usage metrics of the ext-resource service such as node_ext_cpu_usage_seconds_total
3. NodeCpuUsagePromQLFmtStr: sum(count(node_cpu_seconds_total{mode="idle",instance=~"%s.*"}) by (mode, cpu)) - sum(irate(node_cpu_seconds_total{mode="idle",instance=~"%s.*"}[%s])) - (sum(irate(node_ext_cpu_usage_seconds_total{node="%s"}[%s])) or Vector(0))

[shijieqin]

Currently, I collect containers that use ext resources by traversing pods on the kubernetes node, and then get that CPU usage from cAdvisor. Should I get it directly from the kernel(/proc/bt_stat), or go straight past it from cAdvisor(label or cgroup.priority ?).

[kitianFresh]

sum(count(node_cpu_seconds_total{mode="idle",instance=~"%s.*"}) by (mode, cpu)) - sum(irate(node_cpu_seconds_total{mode="idle",instance=~"%s.*"}[%s])) is node current cpu usage already, this is node exporter cpu metrics from the node global, Do not need minus (sum(irate(node_ext_cpu_usage_seconds_total{node="%s"}[%s])) which is ext pods consumed usage again. the former query include all cpu usage of node already which include node_ext_cpu_usage_seconds_total.

I do not understand why you need to minus node_ext_cpu_usage_seconds_total. if you mean the ext pods consumed the predictor prediction available ext resource after ext pods scheduled to node instantly, then the crane-agent can minus it after watched the pods started when patch the node ext resource.

If you just want to use nodes total cpu minus all online pods cpu usage, then minus all offline cpu usages, then it will get all offline cpu resource to be available is what you want. this is only the cpu usage which is kubernetes managed pods, do not include some process that running in node which is not managed by kubernetes.

So node available cpu resource vs node kubernetes ext resource is different. the former make sure we got node's total usage, the latter make sure we got is kubernetes managed resource. I think the two both needed for node qos.

maybe you need give an example and more detailed describe.

[shijieqin]

Maybe I didn't make it clear, and I'll give you an example At first, the cpu of the node was 64 cores, the online service used 32 cores, so the ext-CPU was calculated to be 32000, Offline service A is then scheduled to this node, requesting ext-CPU 16,000, assuming 15 cores are used At this point, if the 15 cores used by the ext-resource service(Offline service A ) are not subtracted, the ext-cpu calculated again is 17000 ((64-32-16)*1000), and the remaining allocatable ext-CPU is 1000 (17000-16000), but the actual idle CPU is 17 cores, so the remaining allocatable ext-CPU should be (32000-16000) in order to schedule more ext-resource services to the node