GoogleCloudPlatform
Prometheus not supporting monitoring on TLS1.3
What did you do?
Upgrading prometheus version from v2.45.3-gmp.1-rc.0 to v2.45.3-gmp.7-rc.0 with prometheus operator and prometheus config reloader keep in version v0.74.0 from prometheus project.
What did you expect to see?
Managed prometheus should be compatible with the other image from prometheus project.
What did you see instead? Under which circumstances?
Seeing the below error when prometheus is scraping targeting prometheus operator.
Get "https://192.168.135.229:10250/metrics": remote error: tls: protocol version not supported
After investigation, we could see that prometheus operator in v0.74.0 is setting the default tls-min-version as VersionTLS13 https://github.com/prometheus-operator/prometheus-operator/blob/v0.74.0/Documentation/operator.md?plain=1#L108-L109 And if we set it to VersionTLS12, the issue will be resolved.
It seems that TLS13 is no longer supported. We think the PR enable boring crypto is the cause. Seems default supported version is only TLS 1.2 when it is using the FIPS compliant one. https://github.com/golang/go/blob/9177e12ccc2115e44de824ae7247ace88617c29a/src/crypto/tls/defaults.go#L93-L95 boring needFIPS() https://github.com/golang/go/blob/master/src/crypto/tls/boring.go#L13 Get supported version with needFIPS() https://github.com/golang/go/blob/9177e12ccc2115e44de824ae7247ace88617c29a/src/crypto/tls/common.go#L1091-L1093
This should be a bug in prometheus that it no longer supports TLS13 which it used to. It will break any monitoring on TLS13 when upgrading to the latest version.
System information
No response
Prometheus version
No response
Prometheus configuration file
No response
Alertmanager version
No response
Alertmanager configuration file
No response
Logs
No response
Thanks for your investigation. There were indeed changes around Boring Crypto as part of our efforts to move to FIPS compliant crypto libraries. However, there is no particular reason not to use TLS 1.2, as far as I know.
When Go supports a different FIPS validated crypto library, I suspect we will be able to move to that and eliminate the limitations around BoringCrypto.
Thanks for the update! We are facing issue due to some software we installed with default manifest that uses TLS1.3 We configured them to use TLS1.2 and it resolved the issue.