[Windows] Support root certificates that are installed lazily

[loic-sharma]

Problem

Windows installs a subset of trusted root certificates lazily. Dart's certificate verification should trigger these root certificates' installation if necessary.

This affects all Let's Encrypt certificates.

See:

1. https://community.letsencrypt.org/t/isrg-root-lazy-loading-problem-missing-from-random-updated-windows-10-versions/141550
2. https://community.letsencrypt.org/t/root-cert-not-included-in-windows-10/177956

Repro

On a brand new Windows box (or use Windows sandbox), run this app:

import 'package:http/http.dart' as http;

void main() async {
  var url = Uri.parse('https://letsencrypt.org/');
  var response = await http.get(url);
  print('Response status: ${response.statusCode}');
  print('Response body: ${response.body}');
}

It results in the following error:

Unhandled exception:
HandshakeException: Handshake error in client (OS Error:
        CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(../../third_party/boringssl/src/ssl/handshake.cc:393))
#0      _SecureFilterImpl._handshake (dart:io-patch/secure_socket_patch.dart:99:46)
#1      _SecureFilterImpl.handshake (dart:io-patch/secure_socket_patch.dart:142:25)
#2      _RawSecureSocket._secureHandshake (dart:io/secure_socket.dart:915:54)
#3      _RawSecureSocket._tryFilter (dart:io/secure_socket.dart:1044:19)
<asynchronous suspension>

Dart SDK Version: Dart SDK version: 2.19.6 (stable) (Tue Mar 28 13:41:04 2023 +0000) on "windows_x64"

Workaround

Use PowerShell to access the affected endpoint:

Invoke-WebRequest https://letsencrypt.org

This causes Windows to install the trusted root certificate. The Dart app will now work as expected without producing a CERTIFICATE_VERIFY_FAILED error.

[a-siva]

//cc @brianquinlan

[alexei-g-aloteq]

It happens for us not only with LetsEncrypt certificates.

Also affected any other certificate chains not loaded onto the Windows trusted store. Namely, calls to something like https://x.cloudfunctions.net (Google cloud) also fails.

But if I try to fetch this address with a curl (like 'curl -v https://x.cloudfunctions.net') - the lazy loading triggers and Windows fetches the root certificate. After that, requests from dart begins to work.

[alexei-g-aloteq]

Hello? Anyone tried to reproduce it?

[penguisnt]

Hi, I'm having the same issue with needing to make requests to Google APIs. My Firebase Auth login was failing, until I called the identitytoolkit API endpoint directly from powershell. Would love to understand what is going on here.

[alexei-g-aloteq]

Another observation.

Many developers do not see this issue with ISRG Root X1 (aka Letsencrypt) certs because have Spotify installed. Being installed (often automatically by Windows), even not being used, Spotify make requests to their servers at least on automatic update installations through the Windows update.

The thing is: Since Spotify uses X1 certificates and triggers the ISRG Root X1 root certificate to load. Then your requests to an endpoint secured with Letsencrypt certificate start to work. Bot not for the others.

Before trying to reproduce this issue, first disable Spotify (better - uninstall), then remove ISRG Root X1 from trusted root CAs folder and try.

If you see that ISRG Root X1 cert is reappeared - it seems that an another application just made a native network request which triggered the lazy root certificate download. It will not happen if you making similar request from Dark runtime.

[ntkme]

Ironically, this even impacts https://pub.dev, that a fresh installation of dart-sdk on GitHub's Hosted Windows on Arm runner fails to run dart pub get out of box. We had to add the following workaround in dart-sass' GitHub Actions:

    - run: Invoke-WebRequest https://pub.dev
      if: runner.os == 'Windows'
      shell: powershell

I think the root of the problem is Microsoft's TLS implementation a.k.a Schannel would lazily initialize the trusted CA certificates, but dart is built with BoringSSL which would only read the already initialized trusted CA.

This isn't an easy thing to fix, unless dart-sdk adds support of using Schannel on Windows, or ships its own default CA bundle like Chrome.

[escamoteur]

Ah, just ran into this too in our case on an accelerate endpoint from S3

[albe-jj]

We run into the same issue. In several windows machines where the root certificate Hellenic Academic and Research Institutions ECC RootCA 2015 is not present by default.

When for instance requesting https://connect.surfconext.nl/.well-known/openid-configuration which requires the Hellenic root certificate the dart http library it will throw error:

CERTIFICATE_VERIFY_FAILED: self signed certificate in certificate chain(../../../flutter/third_party/boringssl/src/ssl/handshake.cc:393))

This error happens because:

1. Upon request three certificates are received from the server: server, intermediate and root certificate
2. It verifies each signature moving up the chain
3. When it reaches the root certificate, it checks if this root certificate is in the Windows trusted Certificate Store
4. Since the certificate is missing and the certificate the issuer and the subject are the same it throws a self signed certificate error (although I believe the error is misleading and it would be better to raise unable to get local issuer certificate error as OP describes)

To reproduce this you can just delete a root certificate from windows certificate store, and send a https request using the dart http library to a server that is linked to that root certificate. To add back the root certificate you can just send the same https request with curl or power shell. If the certificate is installed this way, then https request from dart library will work.

A new issue compared to OP is that here I get error self signed certificate in certificate chain instead of unable to get local issuer certificate which is really confusing because the server certificate is not self signed, only the root certificate is (which is always the case for root CA).

@loic-sharma the issue title is wrong, it should read: [Windows] Support root certificates that are NOT installed lazily But to make it clearer I would suggest to change it to one of the options below

• HTTPS requests fail when root CA certificate is missing from Windows certificate store
• Http library should automatically install root certificates if missing from Windows certificate store