Distributing CA certificates for OpenSSL on Windows

[HertzDevil]

Currently, HTTPS requests do not work on Crystal on Windows:

require "http/client"

HTTP::Client.get("https://example.com") # Unhandled exception: SSL_connect: error:0A000086:SSL routines::certificate verify failed (OpenSSL::SSL::Error)

This is because OpenSSL's own certificate store is empty; we do not build OpenSSL with the --openssldir option, so that directory defaults to C:\Program Files\Common Files\SSL, which contains no certificates, or perhaps the directory does not even exist. There are a few ways to fix this:

• Download cacert.pem here and define the environment variable %SSL_CERT_FILE% to point to it. This workaround should always work even if one of the other options below are chosen. The problem is it could also affect applications using OpenSSL that are not built by Crystal.
• Distribute the above file and call OpenSSL::SSL::Context#ca_certificates= ourselves all the time, but only on Windows. Any executables built by Crystal would use a hardcoded file path, only that the path is relative to the compiler.
• Define --openssldir. This most likely rules out portable installations because that directory ultimately expands to an absolute path that never depends on Crystal's location. RubyInstaller2 does this; Ruby uses C:\Ruby??\ssl, which is distinct from the equivalent directory for the MSYS environment it comes with.
• Use Windows's system certificate store, for example as outlined here. This is the most complex alternative and I don't think it is worth the effort.

[beta-ziliani]

Can this be solved if an installer populates C:\Program Files\Common Files\SSL?

[straight-shoota]

Certificates for OpenSSL on Windows are a notorious problem. I can't imagine we could find an easy solution that just works.

A common mechanism is to distribute a copy with the binary and have the program load that one (instead of looking at a common path or getting help from the OS because that just doesn't work very well).

So I think we can support that use case with providing a compile-time configuration option for a path to the certificates file (which can of course be relative to the binary location). It could have a nice default value. And there should be options to override it at runtime (with %SSL_CERT_FILE% and a custom env var).

Any executables built by Crystal would use a hardcoded file path, only that the path is relative to the compiler.

That wouldn't work for executables that are supposed to be shipped without the compiler, which is one of the most common use cases for everything that's not a development build. So I don't think number 2 can work. Same for number 3. We can't simply do the same thing as Ruby because everyone needs the Ruby interpreter installed to run a Ruby program, but you don't need Crystal compiler installed to run a Crystal program.

[devnote-dev]

@HertzDevil I'm using Crystal 1.7.0 on Windows which fails with a similar error:

Unhandled exception: SSL_connect: error:16000069:STORE routines::unregistered scheme (OpenSSL::SSL::Error)

As you stated, that path does not exist, however after making the SSL directory and saving the cacert.pem file to that location, the error persists. I have to manually configure Crystal to use that file location in my code which then works.

After reading @straight-shoota's comments I understand why using a file location for this isn't the best idea, but what if it were in a location that is guaranteed to exist? I.e., what if Crystal shipped the certificates in the lib directory, then you can be certain that it exists unless a user modifies it. I think this would be a good solution since users shouldn't really be touching anything in that directory, and anything removed from it would likely cause issues when compiling (e.g. removing pcre.lib then trying to use regex).

Alternatively, how about embedding the contents of the file at compile-time? I'm not exactly sure how this would work, but my initial thoughts is something like the read_file macro but obviously on a more technical level.

[HertzDevil]

what if it were in a location that is guaranteed to exist? I.e., what if Crystal shipped the certificates in the lib directory, then you can be certain that it exists unless a user modifies it.

Like I mentioned on Discord, our only Windows distribution is a portable archive rather than a full installer, so the Crystal directory itself can change, but --openssldir does not support relative paths.

[HertzDevil]

So it appears the last option is far easier than I anticipated. Here is a fully working example, using cURL's source code as a reference:

require "openssl"

# lib_crypto.cr
lib LibCrypto
  fun d2i_X509(a : X509*, ppin : UInt8**, length : Long) : X509

  alias X509_STORE = Void*

  fun x509_store_add_cert = X509_STORE_add_cert(ctx : X509_STORE, x : X509) : Int
end

# lib_ssl.cr
lib LibSSL
  fun ssl_ctx_get_cert_store = SSL_CTX_get_cert_store(ctx : SSLContext) : LibCrypto::X509_STORE
end

# wincrypt.cr
@[Link("crypt32")]
lib LibC
  alias HCERTSTORE = Void*
  alias HCRYPTPROV_LEGACY = Void*

  struct CERT_NAME_BLOB
    cbData : DWORD
    pbData : BYTE*
  end

  struct CRYPT_INTEGER_BLOB
    cbData : DWORD
    pbData : BYTE*
  end

  struct CRYPT_OBJID_BLOB
    cbData : DWORD
    pbData : BYTE*
  end

  struct CRYPT_BIT_BLOB
    cbData : DWORD
    pbData : BYTE*
    cUnusedBits : DWORD
  end

  struct CRYPT_ALGORITHM_IDENTIFIER
    pszObjId : LPSTR
    parameters : CRYPT_OBJID_BLOB
  end

  struct CERT_PUBLIC_KEY_INFO
    algorithm : CRYPT_ALGORITHM_IDENTIFIER
    publicKey : CRYPT_BIT_BLOB
  end

  struct CERT_EXTENSION
    pszObjId : LPSTR
    fCritical : BOOL
    value : CRYPT_OBJID_BLOB
  end

  struct CERT_INFO
    dwVersion : DWORD
    serialNumber : CRYPT_INTEGER_BLOB
    signatureAlgorithm : CRYPT_ALGORITHM_IDENTIFIER
    issuer : CERT_NAME_BLOB
    notBefore : FILETIME
    notAfter : FILETIME
    subject : CERT_NAME_BLOB
    subjectPublicKeyInfo : CERT_PUBLIC_KEY_INFO
    issuerUniqueId : CRYPT_BIT_BLOB
    subjectUniqueId : CRYPT_BIT_BLOB
    cExtension : DWORD
    rgExtension : CERT_EXTENSION*
  end

  struct CERT_USAGE
    cUsageIdentifier : DWORD
    rgpszUsageIdentifier : LPSTR*
  end

  X509_ASN_ENCODING   = 0x00000001
  PKCS_7_ASN_ENCODING = 0x00010000

  struct CERT_CONTEXT
    dwCertEncodingType : DWORD
    pbCertEncoded : BYTE*
    cbCertEncoded : DWORD
    pCertInfo : CERT_INFO*
    hCertStore : HCERTSTORE
  end

  fun CertOpenSystemStoreW(hProv : HCRYPTPROV_LEGACY, szSubsystemProtocol : LPWSTR) : HCERTSTORE
  fun CertCloseStore(hCertStore : HCERTSTORE, dwFlags : DWORD) : BOOL

  fun CertEnumCertificatesInStore(hCertStore : HCERTSTORE, pPrevCertContext : CERT_CONTEXT*) : CERT_CONTEXT*
  fun CertGetEnhancedKeyUsage(pCertContext : CERT_CONTEXT*, dwFlags : DWORD, pUsage : CERT_USAGE*, pcbUsage : DWORD*) : BOOL
end

class OpenSSL::X509::Certificate
  def self.from_der(bytes : Bytes)
    ptr = bytes.to_unsafe
    x509 = LibCrypto.d2i_X509(nil, pointerof(ptr), bytes.size)
    raise OpenSSL::Error.new("d2i_X509") unless x509
    {new(x509), bytes[ptr - bytes.to_unsafe..]}
  end
end

module Crystal::System::Crypto
  private ROOT = System.to_wstr("ROOT")

  private def self.each_system_root_certificate(&)
    now = ::Time.utc

    return unless cert_store = LibC.CertOpenSystemStoreW(nil, ROOT)

    cert_context = Pointer(LibC::CERT_CONTEXT).null
    while cert_context = LibC.CertEnumCertificatesInStore(cert_store, cert_context)
      next unless cert_context.value.dwCertEncodingType == LibC::X509_ASN_ENCODING

      next if cert_context.value.pbCertEncoded.nil?

      not_before = Crystal::System::Time.from_filetime(cert_context.value.pCertInfo.value.notBefore)
      not_after = Crystal::System::Time.from_filetime(cert_context.value.pCertInfo.value.notAfter)
      next unless not_before <= now <= not_after

      if LibC.CertGetEnhancedKeyUsage(cert_context, 0, nil, out eku_size) != 0
        eku = Pointer(UInt8).malloc(eku_size).as(LibC::CERT_USAGE*)
        next unless LibC.CertGetEnhancedKeyUsage(cert_context, 0, eku, pointerof(eku_size)) != 0
        next unless (0...eku.value.cUsageIdentifier).any? do |i|
          String.new(eku.value.rgpszUsageIdentifier[i]) == "1.3.6.1.5.5.7.3.1" # serverAuth OID
        end
      end

      encoded = Slice.new(cert_context.value.pbCertEncoded, cert_context.value.cbCertEncoded)
      cert, _rest = OpenSSL::X509::Certificate.from_der(encoded)
      yield cert
    end
  ensure
    LibC.CertCloseStore(cert_store, 0) if cert_store
  end

  private class_getter system_root_certificates : Array(OpenSSL::X509::Certificate) do
    certs = [] of OpenSSL::X509::Certificate
    each_system_root_certificate { |cert| certs << cert }
    certs
  end

  def self.populate_system_root_certificates(ssl_context)
    cert_store = LibSSL.ssl_ctx_get_cert_store(ssl_context)
    system_root_certificates.each do |cert|
      LibCrypto.x509_store_add_cert(cert_store, cert)
    end
  end
end

abstract class OpenSSL::SSL::Context
  protected def initialize(method : LibSSL::SSLMethod)
    previous_def
    Crystal::System::Crypto.populate_system_root_certificates(self)
  end
end

require "http"

puts HTTP::Client.get("https://example.com").body

Here the system root certificates are cached, and added to all constructed OpenSSL::SSL::Contexts except ones created via Context.insecure. This is probably the behavior that most people would expect in a standard library. Feel free to create a PR based on the above code