Container Memory Calculation

[zhiweiv]

I deployed a pod with following yaml

apiVersion: v1
kind: Pod
metadata:
  name: memtest
spec:
  containers:
  - name: test
    image:  mcr.microsoft.com/windows/servercore:ltsc2019
    command: ["PowerShell", "while($true) { sleep 999 }"]

kubectl top pod shows pod memtest used 101Mi memory, however the total memory is more than that, seems only the main process's memory is counted as container memory(powershell.exe 99,196 K), you can see other system processes(mainly svchost.exe) in container used more memory but not counted.

kubectl exec memtest -- tasklist

Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
System Idle Process              0                            0          8 K
System                           4                            0        112 K
smss.exe                      6880                            0      1,232 K
csrss.exe                    10400 Services                  15      5,096 K
wininit.exe                  11796 Services                  15      7,052 K
services.exe                  2184 Services                  15      6,544 K
lsass.exe                     7676 Services                  15     26,676 K
fontdrvhost.exe               7608 Services                  15      3,320 K
svchost.exe                  10496 Services                  15     18,564 K
svchost.exe                   7888 Services                  15     20,892 K
svchost.exe                   1320 Services                  15     69,356 K
svchost.exe                   9992 Services                  15     18,768 K
CExecSvc.exe                 10220 Services                  15      5,164 K
svchost.exe                   6424 Services                  15     24,352 K
svchost.exe                   8772 Services                  15     46,928 K
svchost.exe                  10076 Services                  15     13,928 K
svchost.exe                  11684 Services                  15     53,504 K
svchost.exe                   4208 Services                  15     15,836 K
powershell.exe               10080 Services                  15     99,196 K -- main process
msdtc.exe                     3236 Services                  15      9,948 K
tasklist.exe                 11104 Services                  15      8,184 K
WmiPrvSE.exe                 10860 Services                  15      8,428 K

[zhiweiv]

I get same memory usage result with crictl to query the container stats directly in host, so it should not be the kubernetes issue. I think the value was retrieved from hcsshim via cri interface.

[zhiweiv]

This lead a new issue, seems the node memory usage is based on pods memory usage in it. kubectl top node shows a Windows node 70% memory usage, but the actual usage in Windows task manager is 90%.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[brasmith-ms]

Hi @zhiweiv, thanks for the information here. We're aware of this issue and are working to fix the stats output from containers soon. There are a lot of things in the pipeline for performance and monitoring of Windows containers so I can assure you this will be fixed but I can't guarantee a timeline at the moment.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @brasmith-ms, please provide an update or close this issue.

[fady-azmy-msft]

@Howard-Haiyang-Hao are you familiar with this issue?

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @Howard-Haiyang-Hao, @brasmith-ms, please provide an update or close this issue.

[Howard-Haiyang-Hao]

@zhiweiv, I've reached out to the feature team for an update on this issue. I'll inform you as soon as I receive any updates.

[Howard-Haiyang-Hao]

@zhiweiv Please follow the progress of this issue through the PR: (https://github.com/kubernetes/kubernetes/pull/122999). You'll find updates on when it will be addressed there. Thanks marosset for providing the information.

[zhiweiv]

Thanks for the update, but seems https://github.com/kubernetes/kubernetes/pull/122999 is related to CPU usage of Windows Container, this issue is something with Memory usage.

[jwilsonCX]

Hi @zhiweiv, thanks for the information here. We're aware of this issue and are working to fix the stats output from containers soon. There are a lot of things in the pipeline for performance and monitoring of Windows containers so I can assure you this will be fixed but I can't guarantee a timeline at the moment.

Hi Brandon, your tantalizing account of exciting things in the pipeline "soon" was issued over 1.5 years ago! How long must we wait before we can obtain accurate observability metrics for our Windows container workloads?

Also, please mister/miss Microsoft bot, kindly don't close this thread.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @Howard-Haiyang-Hao, @brasmith-ms, please provide an update or close this issue.