metrics-server with kubenetes 1.24 metrics-server:v0.6.1 not collecting correct memory utilization

Free -m showing less memory utilization but kubectl showing high memory utilization Kubenetes version: 1.24.2 metrics-server: v0.6.1

free -m total used free shared buff/cache available Mem: 7593 3264 160 7 4168 4018

kubectl top no NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 10.231.11.100 750m 18% 5543Mi 76%

These are few pods running on this node

NAMESPACE NAME CPU(cores) MEMORY(bytes) default xxxxxxxxxxxxxx 3m 233Mi default xxxxxxxxxxxxxxx 3m 254Mi default xxxxxxxxxxxxxxxx 370m 909Mi default xxxxxxxxxxxxxxxxxxxx 5m 82Mi default xxxxxxxxxxxxxxxxxxxxx 1m 1Mi default xxxxxxx 4m 89Mi dynatrace xxxxxxxxxxxxxxactivegate-0 66m 712Mi dynatrace xxxxxxxxxxxxxxoneagent-m8sbs 53m 385Mi kube-system kube-flannel-ds-24mtl 3m 20Mi kube-system node-local-dns-pfmns 3m 19Mi xxxxxx xxxxxxx 1m 38Mi

Oct 18 '23 05:10 bpsingh11oct85

/assign @CatherineF-dev /triage accepted

Oct 19 '23 16:10 dashpole

/assign @dgrisonnet

Oct 19 '23 16:10 dashpole

QQ: what is the total node memory? Is it 5543Mi / 76% = 7293?

Oct 19 '23 17:10 CatherineF-dev

8GB

Oct 20 '23 02:10 bpsingh11oct85

QQ: does 8GB mean both free -m and kubectl top no are not accurate?

free -m gave 7593 instead of 8*1024
kubectl top no gave 5543/0.76 = 7293

Oct 23 '23 13:10 CatherineF-dev

free -m looks correct but kubectl top no not looks match with free -m

Oct 24 '23 02:10 bpsingh11oct85

Could you sum up all memory of pods running on the same node? Use kubectl top pods + grep.

What is this value?

Oct 24 '23 22:10 CatherineF-dev

you can see below only 9 pods are running on this node and utilization sum 1116 MB total 37 106 101 1 46 695 31 58 41

1116

kubectl top no |grep -w 10.231.11.21 10.x.x.x 102m 2% 5079Mi 69%

in above matrix server saying 69% utilization of memory and see below OS is saying 5962 is available memory

                        total            used        free      shared    buff/cache       available

Mem: 7593 1322 334 7 5935 5962 Swap: 0 0 0

Oct 27 '23 04:10 bpsingh11oct85

Hi any suggestion on this? kubectl top node saying that 5079 Mi memory is in used but free command shows that that only 1322 Mi is being used which one we should consider ?

Or I am doing mistake to understand this kubectl top node output ?

Nov 05 '23 03:11 bpsingh11oct85

Could you try scraping kubelet summary api metrics, which are the data source for metrics-server?

# ssh into this node
curl -k http://node_ip:10255/stats/summary # port will be 10250 if 10255 is not open

# then check node memory from the above

https://github.com/kubernetes-sigs/metrics-server/blob/master/KNOWN_ISSUES.md

Jan 05 '24 14:01 CatherineF-dev