[WA-31] PDH Checks - Retrying Failed Instance Initializations

[clarkb7]

What does this PR do?

On each check interval try to initialize PDH counters if they are not already initialized.

For the agent to find new counters the PDH object cache needs to be refreshed. The refresh is cached as Windows PDH internally has a mutex on PDH calls, and the refresh is sometimes expensive (order of seconds).

Motivation

Support issues where core checks failed to load because the counters were not available when the agent started.

Additional Notes

Possible Drawbacks / Trade-offs

AGENT 7716 We are currently investigating an issue relating to refreshing the PDH object cache in our python checks.

Describe how to test/QA your changes

1. Install the agent on a Windows OS
2. Use the following powershell to create the registry values that disable the performance counters needed by the winproc, memory, file_handle, and io checks.

   New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\PerfDisk\Performance" -Name "Disable Performance Counters" -PropertyType "DWORD" -Force -Value 1
   New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\PerfOS\Performance" -Name "Disable Performance Counters" -PropertyType "DWORD" -Force -Value 1
   New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\PerfProc\Performance" -Name "Disable Performance Counters" -PropertyType "DWORD" -Force -Value 1
   

3. Restart the agent
4. Check the agent log and/or agent status for warnings about failing to collect counters for the winproc, memory, file_handle, and io checks
5. Run the following powershell to re-enable the performance counters

   New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\PerfDisk\Performance" -Name "Disable Performance Counters" -PropertyType "DWORD" -Force -Value 0
   New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\PerfOS\Performance" -Name "Disable Performance Counters" -PropertyType "DWORD" -Force -Value 0
   New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\PerfProc\Performance" -Name "Disable Performance Counters" -PropertyType "DWORD" -Force -Value 0
   

6. Watch the log for a new Successfully refreshed performance counters! log line
7. Check the agent log and/or agent status to verify that the winproc, memory, file_handle, and io checks are now completing successfully without errors/warnings.

Reviewer's Checklist

• [ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
• [ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
• [ ] A release note has been added or the changelog/no-changelog label has been applied.
• [ ] Changed code has automated tests for its functionality.
• [ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
• [ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
• [ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
• [ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
• [ ] If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
• [ ] If applicable, the config template has been updated.

[bits-bot]

All committers have signed the CLA.