Best Practices to deploy windows_exporter for Kubernetes

[Gerthum]

Hi Guys

Since Prometheus is recommending windows_exporter as an alternative to node_exporter when running Windows Node Pools in Kubernetes. Does anyone have any best practices for running it in this environment.

Things I can think of:

1. Do you need to run it on the Host or can you run it as a Container?
2. Example Dockerfile/pre-built container?
3. Example configuration file?
4. Example Dashboard?

[AMoghrabi]

Any information on this is appreciated. I also have the same question. Thanks!

[ksmuthuus]

I hope https://github.com/prometheus-community/windows_exporter/pull/581 might help to some extent meet your requirement.

[carlpett]

Just to keep the things in as few places as possible, here's a summary of information gathered in other issues and PRs:

Currently it is not possible to run the windows_exporter inside a container and get meaningful results (you can start it, and you'll get some metrics, but they do not reflect the host machine). Windows does not support privileged containers yet, which is a requirement for this to work. So, at this time, you'll need to install the exporter on the host itself.

[Gerthum]

Hi Guys - We've had some success with the following exporter: https://github.com/cloudworkz/kube-eagle It only runs on the Linux nodes but uses kube state metrics to pull in the windows nodes and pods metrics. No need to run it on the windows nodes.

[keikhara]

@Gerthum , when you pull the metrics using kube state metrics, it can show the node and pod metrics but it chokes after certain amount of pod. I would not recommend it. I think it was 30pods or so.

[keikhara]

@carlpett Windows provides named pipe so you can pull information about the pods. https://dille.name/blog/2017/11/29/using-the-docker-named-pipe-as-a-non-admin-for-windowscontainers/ Can you elaborate why you need the privilege container on Windows and what you are trying to do?

[carlpett]

@keikhara The windows_exporter has a much larger scope than just container information. And the rest of the exporter primarily uses the WMI and Performance Counter subsystems in Windows, which are not available from unprivileged containers.

edit: As I see that you work within Microsoft, if you have any nice ideas for how this could be accomplished, I'd be super happy about any tips!

[DerrickMartinez]

For now, you need to run it through wins. This is config for the old version but it still works. Proxy is a simple proxy I wrote in go. Prometheus scrapes the proxy, proxy hits the host's IP. Works very well

total 67680
drwxr-xr-x   8 dmartinez  staff       256 Nov  4 16:44 .
drwxr-xr-x@ 30 dmartinez  staff       960 Nov  5 10:56 ..
-rw-r--r--@  1 dmartinez  staff       426 May 14 14:54 Dockerfile
-rw-r--r--   1 dmartinez  staff       192 Apr 27  2020 entry.ps1
-rwxr-xr-x   1 dmartinez  staff   6361088 Apr 28  2020 proxy.exe
-rw-r--r--   1 dmartinez  staff       131 Apr 27  2020 proxy_entry.ps1
-rw-r--r--@  1 dmartinez  staff  12459520 Apr 24  2020 wins.exe
-rw-r--r--@  1 dmartinez  staff  15818240 Apr 24  2020 wmi_exporter-0.11.1-amd64.exe

proxy_entry.ps1

$j = C:\wins\wins.exe cli net get | ConvertFrom-Json
C:\wins\proxy.exe --remote-port 9182 --remote-ip $j.AddressCIDR.split("/")[0]

Dockerfile

# escape=`
FROM mcr.microsoft.com/powershell:nanoserver-1909
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

COPY wins.exe C:\\wins\\
COPY proxy.exe C:\\wins\\
COPY wmi_exporter-0.11.1-amd64.exe C:\\wins\\wmi-exporter.exe
COPY entry.ps1 entry.ps1
COPY proxy_entry.ps1 proxy_entry.ps1

EXPOSE 9182 8080
USER ContainerAdministrator
CMD ["pwsh", "-f", "entry.ps1"]

entry.ps1

copy c:\wins\wmi-exporter.exe c:\host
c:\wins\wins.exe cli prc run --path c:\host\wmi-exporter.exe --args "--collectors.enabled container,cpu,cs,logical_disk,net,os,system" --exposes TCP:9182

[casey-robertson]

Also for those of us running the prometheus operator, it would be nice to have this integrated.

[mustufa5689]

Hello DerrickMartinez,

Could you help me in creating a WMI exporter container image, that can run as a Daemonsets in my existing K8s clusters and captures all the metrics from the POD level too.

Currently, i have an WMI exporter running as local service on Windows Nodes, to pull metrics from the HOST level, but need WMI exporter or any other metrics agent which can pull the pod/container level metrices from the Windows node in existing prometheus server(running as a pod) and can show on Grafana dashboard (running as a pod). Any help would be appreciated.

Thanks.

[marciogmorales]

@mustufa5689 Were you able to make it happen? I'm looking for the same solution.

[trhumphries]

I'd be interested as well

[auroaj]

As I understand, wins is this: https://github.com/rancher/wins But it requires running service wins on host node, what is not available in case of AKS - it doesn't have any cloudinit option for new nodes, and it can't be replaced with privileged containers because windows host doesn't support it. (correct me if I'm wrong)

[aiyengar2]

FYI, we maintain a Helm chart at Rancher for this: https://github.com/rancher/charts/tree/release-v2.5/charts/rancher-windows-exporter/rancher-windows-exporter/0.1.000.

We package windows_exporter.exe with wins in a Dockerfile (via https://github.com/rancher/windows_exporter-package) and deploy it via the Helm chart.

[aiyengar2]

All the dashboards in https://github.com/rancher/charts/tree/release-v2.5/charts/rancher-monitoring/rancher-monitoring/14.5.100/files/rancher also embed windows_exporter and corresponding node_exporter metrics in a single dashboard.

[mustufa5689]

@mustufa5689 Were you able to make it happen? I'm looking for the same solution.

Yes, I was able to get it worked. There is a yaml file available to deploy windows wmi exporters inside K8s cluster running as Daemonsets.

Also, you need Rancher wins to place on the host: https://github.com/rancher/wins

I have place the file in text format, you can download and rename to yaml. wmi-exporter.txt

It worked properly on my RKE cluster. For other K8s distributions and managed K8s services should work.

Steps:-

1. Run the yaml file attached
2. Run below commands on windows node,

[powershell]
Invoke-WebRequest -Uri https://github.com/rancher/wins/releases/download/v0.0.4/wins.exe -OutFile C:\wins.exe
write-output cmd` /c` start` /b` c:\wins.exe` srv` app` run | Out-File C:\wins.ps1
New-Service -Name TestService -BinaryPathName C:\wins.exe` srv` app` run
Start-Service -Name TestService

3. This should make your pod up and running. --- Check the logs of the wmi_exporter pods.

Prometheus will be able to scrap metrics from the windows wmi_exporter agents so no need to add the datasource on grafana dashboard.

Thanks, Mustufa M.

[alfianabdi]

I run the windows_exporter as service inside the worker node ( not using daemon-set ).

$url = "https://github.com/prometheus-community/windows_exporter/releases/download/v0.14.0/windows_exporter-0.14.0-amd64.msi"
$output = "C:\Users\windows.msi"
$start_time = Get-Date
$wc = New-Object System.Net.WebClient
$wc.DownloadFile($url, $output)
msiexec /i C:\Users\windows.msi ENABLED_COLLECTORS=cpu,cs,logical_disk,net,os,service,system,textfile,memory

Then using the node role of the kubernetes_sd_config

  - job_name: 'kubernetes-windows-exporter'
    scheme: http
    kubernetes_sd_configs:
      - role: node
        selectors:
          - role: node
            label: 'kubernetes.io/os=windows'
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels: [__address__]
        regex: (.+):(.+)
        target_label: __address__
        replacement: $1:9182

[a-mazed]

Excuse me, but I would like to know when this pr will be implemented? We want to use windows_exporter in k8s

[srjithsn]

@aiyengar2 I was able to install https://github.com/rancher/charts/tree/release-v2.5/charts/rancher-windows-exporter/rancher-windows-exporter/0.1.000 in AWS EKS. It's a test environment. Host wins version is v0.1.3 and pod version is 0.1.0.

Pods are up and running. Prometheus Identified the Target. But metrics are not getting published when accessing the service endpoint. I am getting output from host(k8s node) while trying to do curl. No response from inside the pod.

Is there anything that I have missed?

From pod 'exporter-node':

Pod Logs:

From Host:

[jsturtevant]

could likely close this issue. With hostprocess container this is easier and has some docs. There are also examples on wiring all this up with the rest of the Prometheus stack in https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/windows.md