TLS handshake with IfW API on host 'localhost'

[drapiti]

Receiving this error still on plugin output of a few servers, using both 1.12.3 and 1.13.0 beta.

TLS handshake with IfW API on host 'localhost' (SNI: 'rsmqspus01.xxxxx') port '5668' failed: stream truncated [asio.ssl.stream:1]

[LordHepipud]

Could it be possible that the Icinga Agent certificate is not valid or wasn't signed properly during the initial setup? This could cause some issues. What happens if you tried to re-create the Icinga Agent certificate?

[drapiti]

Could it be possible that the Icinga Agent certificate is not valid or wasn't signed properly during the initial setup? This could cause some issues. What happens if you tried to re-create the Icinga Agent certificate?

On certain systems we’ve retried to reinstall the agent/certificate and this maybe fixes 30/40% of the cases but not all. Also normal checks/non powershell framework checks work fine on the same systems.

[ispmonsupporto]

I am experiencing the same issues reported by Drapiti in the already open issue.

The error I am encountering is:

TLS handshake with IfW API on host 'localhost' (SNI: 'sbwmop01.xxxxxx') port '5668' failed: stream truncated [asio.ssl.stream:1] I have tried reinstalling the agent multiple times, but the checks remain in an unknown state. Additionally, I have verified that the certificate is correctly placed in the Trusted Root Certification Authority, and it is present.

I have also noticed some errors in the logs, which I am sharing below:

############################################ _[03/07/2025 11:01:14] Failed to securely establish a communication between this server and the client

A client connection could not be established to this server. This issue is mostly caused by using Self-Signed/Icinga 2 Agent certificates for the server and the client not trusting certificates signed by your trusted CA or setup the client to accept untrusted certificates

Icinga for Windows exception report:

Exception Message: Exception calling "AuthenticateAsServer" with "4" argument(s): "A call to SSPI failed, see inner exception."

Command Origin: Internal

Script Line Number: 32933

Exact Position: At C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache\framework_cache.psm1:32933 char:9 $SSLStream.AuthenticateAsServer($Certificate, $false, $TLSPro ... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

StackTrace: at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception) at New-IcingaSSLStream(Closure , FunctionContext )

Call Stack:

Command Arguments

---

Get-IcingaExceptionString {ExceptionObject=Exception calling "Authentic... Write-IcingaEventMessage {EventId=1500, Namespace=Framework, Exception... New-IcingaSSLStream {Client=System.Net.Sockets.TcpClient, Certifi... Open-IcingaTCPClientConnection {Client=System.Net.Sockets.TcpClient, Certifi... New-IcingaForWindowsRESTApi {Port=5668, CertFilter=, Address=, RequireAut...

Object details:

Available : LocalEndPoint : RemoteEndPoint : Handle : 720 Blocking : True UseOnlyOverlappedIO : False Connected : False AddressFamily : InterNetwork SocketType : Stream ProtocolType : Tcp IsBound : True ExclusiveAddressUse : ReceiveBufferSize : SendBufferSize : ReceiveTimeout : SendTimeout : LingerState : NoDelay : Ttl : DontFragment : MulticastLoopback : EnableBroadcast : DualMode :_ ############################################

Could you please provide support in resolving this issue?

Thank you in advance.

regards

[drapiti]

@LordHepipud so finally we have a probable cause. It seems that the servers which are not working have a different case in the hostname, so most likely it is not matching with the fqdn which we force lowercase on agent install. Icinga itself comunicates correctly however the icinga powershell framework does not because it may be requiring the actual case which is configured on the specific server. To resolve this issue I would suggest to provide an alias on certificate setup which includes both names in lower and uppercase. Could you please check this?

[LordHepipud]

Thank you for the details. I have tried to reproduce this behavior, but without any success. The upper/lowercase characters shouldn't count in this scenario.

To test this, I have created an Icinga Agent certificate with a UPPER case hostname and registered that as the Agents name.

Now the certificate does look like this on inspect:

Icinga for Windows Certificate:

Issuer  => CN=Icinga CA
Subject => CN=DEVWIN2022

By checking the hostname, we can see the result:

PS> Get-IcingaHostname
devwin2022

PS> Get-IcingaHostname -ReadConstants
DEVWIN2022

For Icinga for Windows this doesn't matter. The Agent is connecting to the parent node just fine, while the Agent actively calls the Icinga for Windows API directly.

For the moment I have absolutely no clue, on how I can reproduce this issue.

[LordHepipud]

Just out of curiosity - when you re-create the Icinga for Windows certificate by using

Start-IcingaForWindowsCertificateThreadTask;

and restart Icinga for Windows about a minute later - does this issue still persist?

[drapiti]

Start-IcingaForWindowsCertificateThreadTask;

Unfortunately yes, there is no change. What I have noticed is that the majority of servers have OS version < 2016, not all but most with this issue.

[drapiti]

Another thing to note on all WIndows server 2012/r2 servers, the icinga for windows event log is full of 1500 events. But this is true for agents which seem to be working (without TLS errors on the checks) and also those with TLS errors. How is it possible to manually force to use a signed certificate? These 1500 events do not seem to be present on >= Windows Server 2016 using the exact same install procedure. There is definatley something not working on Windows Server 2012 servers with the certificate signing or registration.

[LordHepipud]

Are we talking about 2012 or 2012R2? I will have a closer look on these systems. Haven't noticed anything there yet, but maybe a fresh install would be required to reproduce this issue.

In general you can use the arguments for the Start-IcingaWindowsRESTApi daemon, to force specific certificates being used.

[drapiti]

Are we talking about 2012 or 2012R2? I will have a closer look on these systems. Haven't noticed anything there yet, but maybe a fresh install would be required to reproduce this issue.

In general you can use the arguments for the Start-IcingaWindowsRESTApi daemon, to force specific certificates being used.

As far as I can tell 2012 and 2012 r2 have similar behaviour. From 2016 and up we have less problems.

[LordHepipud]

Thank you for the feedback. I will have a look on this!

[drapiti]

Some additional info which may point us in the correct direction. I think that most of this issue stems from Cipher suite configured on certain machines. Icinga uses the following as reported on Inspect of the curl call:

ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-CHACHA20-POLY1305
DHE-RSA-AES128-GCM-SHA256

On a specific Windows Server 2012 machine where they have enabled the following SSL Cipher Suite Order, none of the above Ciphers match an available Cipher on the system.

Present are:

TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256,
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384,
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P521,
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256,
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384,
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P521,
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256,
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384,
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P521,
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384,
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P521,
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256,
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384,
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P521,
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384,
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P521,
TLS_RSA_WITH_AES_256_GCM_SHA384,
TLS_RSA_WITH_AES_128_GCM_SHA256,
TLS_RSA_WITH_AES_256_CBC_SHA256,
TLS_RSA_WITH_AES_128_CBC_SHA256,
TLS_RSA_WITH_AES_256_CBC_SHA,
TLS_RSA_WITH_AES_128_CBC_SHA,
TLS_RSA_WITH_NULL_SHA256,
TLS_RSA_WITH_NULL_SHA

Typically the event viewer will be full of these events:

Log Name: System Source: Schannel Date: 28/03/2025 12:43:49 Event ID: 36874 Task Category: None Level: Error Keywords:
User: SYSTEM Computer: SBWSTP03.xxx.xxxxx.xxxx.xxx Description: An TLS 1.2 connection request was received from a remote client application, but none of the cipher suites supported by the client application are supported by the server. The SSL connection request has failed. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Schannel" Guid="{1F678132-5938-4686-9FDC-C8FF68F15C85}" /> <EventID>36874</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x8000000000000000</Keywords> <TimeCreated SystemTime="2025-03-28T11:43:49.464027100Z" /> <EventRecordID>1268219</EventRecordID> <Correlation ActivityID="{B6398C25-6D64-4939-A934-41242E54F6EC}" /> <Execution ProcessID="2176" ThreadID="17384" /> <Channel>System</Channel> <Computer>SBWSTP03.xxx.xxx.xxxx.xxx</Computer> <Security UserID="S-1-5-18" /> </System> <EventData> <Data Name="Protocol">TLS 1.2</Data> </EventData> </Event>

Also I think some windows server 2012 servers require the following patch: (https://support.microsoft.com/en-us/topic/update-adds-new-tls-cipher-suites-and-changes-cipher-suite-priorities-in-windows-8-1-and-windows-server-2012-r2-8e395e43-c8ef-27d8-b60c-0fc57d526d94)

[LordHepipud]

This is very interesting, thank you for this feedback. Indeed, my cipher list on my Windows 2012R2 test machine looks different than yours, which might explain why I'm not running into these kinds of issues. I will play around with this more and check how we can resolve this or at least test for this.