Negative timestamps in probe_ssl_last_chain_expiry_timestamp_seconds

[bgagnon]

blackbox_exporter version:

v0.17.0

What did you do that produced an error?

Replaced probe_ssl_earliest_cert_expiry with probe_ssl_last_chain_expiry_timestamp_seconds in our alert definitions to fix issue #340

What did you expect to see?

A positive timestamp, equal to probe_ssl_earliest_cert_expiry in most cases.

What did you see instead?

Value -62135596800 everywhere -- the value does not seem to ever change, unlike probe_ssl_earliest_cert_expiry. The formatted version of that timestamp is 1677-09-21 00:12:43.146 +0000 UTC.

[bgagnon]

The observed value is likely the uninitialized value for time.Time{} when there are zero verified chains:

func getLastChainExpiry(state *tls.ConnectionState) time.Time {
	lastChainExpiry := time.Time{}
	for _, chain := range state.VerifiedChains {
		earliestCertExpiry := time.Time{}
		for _, cert := range chain {
			if (earliestCertExpiry.IsZero() || cert.NotAfter.Before(earliestCertExpiry)) && !cert.NotAfter.IsZero() {
				earliestCertExpiry = cert.NotAfter
			}
		}
		if lastChainExpiry.IsZero() || lastChainExpiry.After(earliestCertExpiry) {
			lastChainExpiry = earliestCertExpiry
		}

	}
	return lastChainExpiry
}

Perhaps the problem is that the probe is related to the tls_config?

tls_config:
  insecure_skip_verify: true

[brian-brazil]

That doesn't seem right, the uninitialised time would be the start of 1970. Have you checked all the returned certs to see if one has a weird expiration?

[bgagnon]

From https://golang.org/pkg/time/#Time:

The zero value of type Time is January 1, year 1, 00:00:00.000000000 UTC. As this time is unlikely to come up in practice, the IsZero method gives a simple way of detecting a time that has not been initialized explicitly.

Still, it does not explain the weird time in the year 1677. I'll check the certs in the chain to see if I can spot a weird expiration date...

[bgagnon]

I ran it through the debugger and can confirm the following when insecure_skip_verify is true:

• state.VerifiedChains is empty
• lastChainExpiry retains its uninitialized value of time.Time{}
• once converted through Unix() and float64(), this yields -62135596800

If I set insecure_skip_verify to false:

• state.VerifiedChains is non empty
• lastChainExpiry gets set to a correct value of 2024-10-02 18:46:05 +0000
• once converted through Unix() and float64(), this yields 1727894765

[bgagnon]

My conclusion is that:

• state.PeerCertificates is always available
• state.VerifiedChains is available only when insecure_skip_verify is false
• probe_ssl_last_chain_expiry_timestamp_seconds should not be given a value when state.VerifiedChains is unavailable

[brian-brazil]

Huh, so Go doesn't even try to verify if verification is disabled?

Considering that by specifying insecure_skip_verify that you don't care about certs, what exactly is it that you're trying to test here?

[bgagnon]

what exactly is it that you're trying to test here?

I don't care that the container in which blackbox_exporter runs does not have the exact same CA bundle(s) that a normal client (Windows, Mac, Linux) running a typical browser (Chrome, Firefox, IE) would have, because those are subject to corporate policies that distribute custom CAs across the organization.

However, I do care about the dates on the certificate(s) presented by the target, because those are definitely going to break users if they are in the past, even if they have the proper CA bundle.

Which was a fine strategy for the old metric probe_ssl_earliest_cert_expiry, but not with probe_ssl_last_chain_expiry_timestamp_seconds which requires validation.

[brian-brazil]

I don't see any way in which that could be made to work. We can only verify chains if chains exist, and that depends on doing the full TLS checking and thus knowing the CAs to apply. Using probe_ssl_last_chain_expiry_timestamp_seconds with insecure_skip_verify is not sane. I'd suggest providing explicit CAs to the exporter.

[bgagnon]

Agreed.

Now, shouldn't probe_ssl_last_chain_expiry_timestamp_seconds be excluded from the metrics when verification is off?

[brian-brazil]

I'm not sure offhand. The PR doesn't seem right though, we shouldn't be poking at the configuration handled by library - the blackbox exporter itself knows nothing about insecure_skip_verify.

[bgagnon]

I updated the PR to use IsZero() as the condition instead of poking in the module configuration.

[candlerb]

@bgagnon:

Still, it does not explain the weird time in the year 1677

According to Ruby, that value corresponds to the date 0001-01-01

irb(main):001:0> Time.at(-62135596800)
=> 0001-01-01 01:00:00 +0100

FWIW, I saw the same issue today as well (hence arriving at this issue). The problem went after a series of changes, and one of those was that I had insecure_skip_verify: true initially, and then changed it to verify proper signed certs from a local CA.

I agree with your conclusion:

probe_ssl_last_chain_expiry_timestamp_seconds should not be given a value when state.VerifiedChains is unavailable

[rosaLux161]

What's the current status?

[RicardoS840]

Hi guys, any news on this?

I'm getting this same error (negative expiration date on the probe_ssl_last_chain_expiry_timestamp_seconds) using both certificates with and without the chain and there are no weird expiration dates on the root or intermediate certificate (I have checked).

[candlerb]

Are you using insecure_skip_verify: True? If so, you'll see this value.

As a workaround, filter out this value in your PromQL expressions:

probe_ssl_last_chain_expiry_timestamp_seconds != -62135596800

or more simply (assuming you don't have any certificates dating to before 1970):

probe_ssl_last_chain_expiry_timestamp_seconds > 0

[candlerb]

@bgagnon: I updated the PR to use IsZero() as the condition instead of poking in the module configuration.

That looks sensible to me: the zero value of this metric is not meaningful, so just don't emit that value if seen. Alternatively, a NaN could be emitted instead.

[RicardoS840]

Hi @candlerb thanks for replying. I will take a look at it and will let you know Regards

[RicardoS840]

Hi @candlerb I do have skip_tls_verify set to true. If I set it to false the probes fail. Why is this? The domains that I'm probing have TLS self signed certificates recognized by my windows machine (even though I'm running the exporter in docker and I think docker is running in WSL and there I haven't add the CA root certs). The probes are working for the certificates expiration dates, I would just like to also get the root certificate expiration date as well.

Cheers

[candlerb]

Hi @candlerb I do have skip_tls_verify set to true. If I set it to false the probes fail. Why is this?

Because the client (blackbox_exporter) thinks that the certificate presented by the server is invalid. This could be because:

1. It's not signed by a Certificate Authority which is known to, and trusted by, the client
2. The certificate's SubjectAltName does not match the name which the client expects (i.e. that it connected to)
3. It has expired

The domains that I'm probing have TLS self signed certificates

That will be why - i.e. case (1) above.

Instead of using insecure_skip_verify (which disables all validation of the certificate), you can use

tls_config:
  ca_file: /path/to/servercert.pem

where servercert.pem contains the server's own certificate. (Actually, what's needed here is the CA's certificate, but since it's self-signed, the server cert and the CA cert are one and the same)

If each server has its own self-signed cert, then you'll need to define a separate blackbox_exporter module config for each server.

You're then left with the issue of the certificate name. If you're connecting to the host as "example.com" or "1.2.3.4", but the certificate contains identity "localhost" (say), then validation will also fail, under point (2) above. You can get around this by specifying the identity you expect to see in the certificate:

tls_config:
  ca_file: /path/to/servercert.pem
  server_name: localhost

Note however that the certificate must contain at least one matching SubjectAltName. If it contains only CommonName (CN), and no SubjectAltName, then the validation will fail. Older versions of Go would also allow matching on CommonName, but this was disabled in go 1.15, and completely removed in go 1.17.

The probes are working for the certificates expiration dates, I would just like to also get the root certificate expiration date as well.

That doesn't make any sense. If the certificate is self-signed, then the certificate is its own root.

[RicardoS840]

Hi @candlerb, sorry I didn't explained properly. I have created a Self signed root certificate and then I use that certificate to sign other certificates that I have also created.

[RicardoS840]

I'm using the same root certificate to all my domains so I believe I can just add it like you've said:

[RicardoS840]

I will try it and will let you know =) Thank you very much for all the support!

[candlerb]

I'm using the same root certificate to all my domains so I believe I can just add it like you've said:

Yes, ca_file: /path/to/cacert.pem where cacert.pem is the home-made root certificate you used to sign other certificates.

[RicardoS840]

Hi @candlerb Good news =) It's working!

Thank you very much for the help ;) Regards