windows_exporter icon indicating copy to clipboard operation
windows_exporter copied to clipboard

terminal_services collector on classic machine

Open arnaudcharles opened this issue 2 months ago • 15 comments

Problem Statement

I'm using the terminal_services collector as per #2114

Windows Exporter is installed on a lot of instance, thanks for your work btw, and on all of them I can see this log : WARNING err="failed collecting terminal services session count metrics: failed to collect Terminal Services Session metrics: performance counter not initialized.

I use this metric on several host to show on Grafana how much RDP session are open and who is connected where. It eas the purpose for us of the logon collector.

Is there a way to not show this warning in the logs, I have one every 30s or should I consider not using the terminal_services collector and find another way like a process, logs or whatever to monitor simple RDP sessions ?

I'm looking for you advice on this, I don't know if this is a bug related to the collector.

Environment

  • windows_exporter Version: 0.31.3
  • Windows Server Version: 2019

arnaudcharles avatar Oct 14 '25 15:10 arnaudcharles

On Application start, there should be raised an error, why the collector can't be initialized

Workaround: --log.level=error

jkroepke avatar Oct 14 '25 19:10 jkroepke

Thanks for your answer @jkroepke. I'm using a config file so I did this :

PS C:\Program files\windows_exporter> .\windows_exporter.exe --config.file="C:\Program Files\windows_exporter\config.yaml" --log.level=error

The command doesn't give or show any output so I ran debug instead and it give me this time=2025-10-14T20:24:10.207Z level=WARN source=collection.go:248 msg="couldn't initialize collector" err="error build collector terminal_services: failed to create Terminal Services Session collector: failed to initialize collector: failed to add counter \\Terminal Services Session(*)\\Handle Count: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Page Faults/sec: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Page File Bytes: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Page File Bytes Peak: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\% Privileged Time: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\% Processor Time: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\% User Time: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Pool Nonpaged Bytes: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Pool Paged Bytes: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Private Bytes: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Thread Count: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Virtual Bytes: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Virtual Bytes Peak: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Working Set: The specified object was not found on the computer.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Working Set Peak: The specified object was not found on the computer.\r\n"

I have the feeling, that this collector is intended to be used with Terminal Services counters meaning all the relative roles or features need to be installed which is not my case. Is it correct ?

Can't we use this collector to simply get RDP metrics (for management purpose) ?

I'm able to get metrics on Prometheus and so grafana using this collector but all these warning are quite troubling.

arnaudcharles avatar Oct 14 '25 20:10 arnaudcharles

Before just fixing something, I would like to get the root cause first: Why the counters are not available?

It could be:

  • ~~Not available at Windows Server 2019~~ (I found an old issuer with an dump of all counter on Windows Server 2019 - the counter was present)
  • Not available because of missing role
  • An local error (e.g. broken counters)

Sadly, I do not have access to an Server 2019 environment.

jkroepke avatar Oct 15 '25 06:10 jkroepke

I also have a similar config on 2022 where the warning appears same as 2019.

I checked some metric and on all of them I can find the metric that I wanted which is windows_terminal_services_session_info BUT other metrics like windows_terminal_services_cpu_time_seconds_total or windows_terminal_services_private_bytes seems not there. Is it possible that this metric cannot be created and then generate the Warning ? What I'm missing to get the full metrics, what are the prerequisite?

arnaudcharles avatar Oct 15 '25 12:10 arnaudcharles

What I'm missing to get the full metrics, what are the prerequisite?

I would also like to solve this question as well. In my demo environments, it just works.

jkroepke avatar Oct 15 '25 13:10 jkroepke

I tried on a few and the only machine I see this kind of behavior on is a Core machine (not Desktop Experience).

Sixty502 avatar Nov 17 '25 22:11 Sixty502

Accordings to Microsoft docs, the dll is availible on Core systems.

https://learn.microsoft.com/en-us/previous-versions/windows/desktop/legacy/ee391633%28v%3Dvs.85%29?utm_source=chatgpt.com

I do not have an system to test this.

But I will check, if the perf counter are there.

jkroepke avatar Nov 18 '25 15:11 jkroepke

I get the same error spammed in eventlog on multiple Windows Server 2022 21H2

On servers without RDSH role:

source=collect.go:220 msg="collector terminal_services failed after 891.6µs, resulting in 20 metrics" err="failed collecting terminal services session count metrics: failed to collect Terminal Services Session metrics: performance counter not initialized. Check application logs from initialization pharse for more information"

On servers with RDSH role:

source=collect.go:220 msg="collector terminal_services failed after 6.9325ms, resulting in 50 metrics" err="failed collecting terminal services session count metrics: failed to collect Terminal Services Session metrics: performance counter not initialized. Check application logs from initialization pharse for more information"

Perhaps could the problem be localization related as suggested in https://github.com/grafana/alloy/issues/4393 ?

deajan avatar Nov 21 '25 15:11 deajan

Check application logs from initialization pharse for more information

I need the logs from initialization

jkroepke avatar Nov 21 '25 15:11 jkroepke

Sure:

C:\Windows\system32>"C:\Program Files\windows_exporter\windows_exporter.exe" --config.file="C:\Program Files\windows_exporter\config.yaml" --collectors.enabled [defaults],cpu_info,terminal_services,memory,tcp,textfile,service,time
time=2025-11-21T16:33:31.333+01:00 level=INFO source=main.go:153 msg="using configuration file: C:\\Program Files\\windows_exporter\\config.yaml"
time=2025-11-21T16:33:31.406+01:00 level=INFO source=textfile.go:111 msg="textfile directories: C:\\Program Files\\windows_exporter\\textfile_inputs" collector=textfile
time=2025-11-21T16:33:31.778+01:00 level=INFO source=net.go:288 msg="nic/addresses collector is in an experimental state! The configuration and metrics may change in future. Please report any issues." collector=net
time=2025-11-21T16:33:31.952+01:00 level=WARN source=collection.go:248 msg="couldn't initialize collector" err="error build collector terminal_services: failed to create Terminal Services Session collector: failed to initialize collector: failed to add counter \\Terminal Services Session(*)\\Handle Count: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Page Faults/sec: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Page File Bytes: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Page File Bytes Peak: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\% Privileged Time: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\% Processor Time: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\% User Time: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Pool Nonpaged Bytes: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Pool Paged Bytes: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Private Bytes: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Thread Count: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Virtual Bytes: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Virtual Bytes Peak: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Working Set: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n\nfailed to add counter \\Terminal Services Session(*)\\Working Set Peak: L’objet spécifié n’a pas été trouvé sur l’ordinateur.\r\n"
time=2025-11-21T16:33:31.954+01:00 level=INFO source=main.go:277 msg="Running as DOMAIN\redacted"
time=2025-11-21T16:33:31.956+01:00 level=INFO source=main.go:190 msg="Enabled collectors: cpu, memory, logical_disk, physical_disk, net, os, service, system, cpu_info, terminal_services, memory, tcp, textfile, service, time"
time=2025-11-21T16:33:31.956+01:00 level=INFO source=main.go:208 msg="starting windows_exporter in 624.1845ms" version=0.31.3 branch=HEAD revision=fcf21bb6008679edab438661a0e882c0c7f27386 goversion=go1.25.1 builddate=20250906-06:52:57 maxprocs=4
time=2025-11-21T16:33:31.958+01:00 level=INFO source=tls_config.go:347 msg="Listening on" address=[::]:9182
time=2025-11-21T16:33:31.958+01:00 level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9182
time=2025-11-21T16:33:43.711+01:00 level=INFO source=main.go:237 msg="Shutting down windows_exporter via kill signal"
time=2025-11-21T16:33:43.711+01:00 level=INFO source=main.go:261 msg="windows_exporter has shut down"

Fetched from a french localized server.

deajan avatar Nov 21 '25 15:11 deajan

Pretty sure the performance counters are actually localized:

C:\Windows\system32>typeperf "\Terminal Services Session\% User Time"

                         z patienter...
Erreur : aucun compteur valide.

C:\Windows\system32>typeperf "\Services Terminal Server\Nb de sessions actives"

"(PDH-CSV 4.0)","\\REDACTED-COMPUTER\Services Terminal Server\Nb de sessions actives"
"11/21/2025 16:41:39.733","1.000000"
"11/21/2025 16:41:40.738","1.000000"
"11/21/2025 16:41:41.743","1.000000"
                         z patienter...
L’opération s’est bien déroulée.
Image

deajan avatar Nov 21 '25 15:11 deajan

Pretty sure the performance counters are actually localized

performance counters are localized, but windows_exporter is using a language-neutral Win32 API to mitigate the issue:

https://learn.microsoft.com/en-us/windows/win32/api/pdh/nf-pdh-pdhaddenglishcounterw

jkroepke avatar Nov 21 '25 15:11 jkroepke

I ask myself, what is the dependency against Terminal Services Session performance counters? It seems that Terminal Services are always present, but per-session counters not.

I guess the server role Remote Desktop Session Host could be the root cause

jkroepke avatar Nov 21 '25 15:11 jkroepke

performance counters are localized, but windows_exporter is using a language-neutral Win32 API to mitigate the issue:

https://learn.microsoft.com/en-us/windows/win32/api/pdh/nf-pdh-pdhaddenglishcounterw

Good to know ;) I noticed that windows_exporter tries to fetch alot of counters which don't even exist on my windows 2022 21H2 RDSH setup. It looks like I only get 3 poor counters. Please find attached the output of Get-Counter -ListSet *

perfcounters.zip

Willing to do some tests if needed.

deajan avatar Nov 21 '25 16:11 deajan

I guess the server role Remote Desktop Session Host could be the root cause

Here's the installed roles on my server that runs Remote Desktop Session Host (RDSH) and that misses said counters:

Get-WindowsFeature | Where-Object -FilterScript { $PSItem.Name -match 'remote'}

Display Name                                            Name                       Install State
------------                                            ----                       -------------
[ ] Accès à distance                                    RemoteAccess                   Available
[X] Services Bureau à distance                          Remote-Desktop-Services        Installed
[ ] Assistance à distance                               Remote-Assistance              Available
            [ ] Outils de chiffrement de lecteur Bit... RSAT-Feature-Tools-B...        Available
        [ ] Outils de gestion de l’accès à distance     RSAT-RemoteAccess              Available
            [ ] Interface GUI de l’accès à distance ... RSAT-RemoteAccess-Mgmt         Available
            [ ] Module d’accès à distance pour Windo... RSAT-RemoteAccess-Po...        Available

Perhaps it would just be sufficient to report those missing counters when logging is set to debug ?

deajan avatar Nov 21 '25 16:11 deajan