[Bug] Running the test runner against a Storybook server using self-signed SSL certs fails

[KrofDrakula]

Describe the bug

When using SSL with a self-signed certificate to run the development server, the test runner (with at least the Chromium browser) will reject the SSL connection, probably because it fails to validate the CA signing the certificate (that's my educated guess).

The test runner needs an option to either inject the root CA into the running instance of the browser, or disable verifying SSL certificates when connecting through HTTPS.

Steps to reproduce the behavior

To test this, generate a CA cert and a corresponding domain certificate. For convenience, I've added two bash scripts to generate the CA and domain wildcard self-signed certs to enable SSL. Not that you will need to add the root.pem certificate to your trusted root CA store in order to be able to use certificates generated by this CA.

generate-ca

#!/bin/bash

set -eu

ROOT_DIR=$(dirname $0)
BUILD_DIR="$ROOT_DIR/build"

mkdir -p "$BUILD_DIR"

openssl genrsa -out "$BUILD_DIR/root.key" 2048
openssl req -x509 -new -nodes -key "$BUILD_DIR/root.key" -sha256 -days 365 -out "$BUILD_DIR/root.pem"
openssl x509 -text -noout -in "$BUILD_DIR/root.pem"

generate-domain-cert

#!/bin/bash

set -e

ROOT_DIR=$(dirname $0)
BUILD_DIR="$ROOT_DIR/build"

ROOT_CERT="$BUILD_DIR/root.pem"
ROOT_KEY="$BUILD_DIR/root.key"

if [ ! -f "$ROOT_CERT" ]
then
  echo "Root certificate not found! Use generate-ca to generate a root CA certificate"
  exit 1
elif [ ! -f "$ROOT_KEY" ]
then
  echo "Root certificate key not found! Use generate-ca to generate a root CA certificate"
  exit 1
fi

if [ "$1" = "" ]
then
  echo "You must provide a FQDN name, e.g. ./generate-domain-name domain.name"
  exit 2
fi

DOMAIN_NAME="$1"
DOMAIN_DIR="$BUILD_DIR/$DOMAIN_NAME"

mkdir -p "$DOMAIN_DIR"

DOMAIN_KEY="$DOMAIN_DIR/wildcard.key"
DOMAIN_CERT="$DOMAIN_DIR/wildcard.crt"
DOMAIN_CSR="$DOMAIN_DIR/wildcard.csr"
DOMAIN_CONFIG_NAME="$DOMAIN_DIR/config.cnf"

openssl genrsa -out "$DOMAIN_KEY" 2048

cat <<EOF > "$DOMAIN_CONFIG_NAME"
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = $DOMAIN_NAME
[v3_req]
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
IP.1 = 127.0.0.1
DNS.1 = localhost
DNS.2 = *.localhost
DNS.3 = $DOMAIN_NAME
DNS.4 = *.$DOMAIN_NAME
EOF

openssl req -new -out "$DOMAIN_CSR" -key "$DOMAIN_KEY" -config "$DOMAIN_CONFIG_NAME"

openssl x509 -req -in "$DOMAIN_CSR" \
-CA "$ROOT_CERT" \
-CAkey "$ROOT_KEY" \
-CAcreateserial \
-out "$DOMAIN_CERT" \
-days 365 \
-sha256 \
-extensions v3_req \
-extfile "$DOMAIN_CONFIG_NAME"

openssl verify -CAfile "$ROOT_CERT" "$DOMAIN_CERT"

openssl x509 -text -noout -in "$DOMAIN_CERT"

Run the scripts:

./generate-ca
./generate-domain-cert mydomain.com

Make sure that mydomain.com resolves to the IP of the machine running the Storybook server.

Then run the Storybook dev server together with the test runner:

concurrently -k -s first \
  "start-storybook -p 6006 -h mydomain.com --https --ssl-ca ./build/root.pem --ssl-cert ./build/mydomain.com/wildcard.crt --ssl-key ./build/mydomain.com/wildcard.key" \
  "wait-on tcp:6006 && STORYBOOK_URL=https://mydomain.com:6006 yarn test-storybook"

Verify that the Storybook is reachable by issuing CURL or by opening the URL in a browser (this verifies your system trusts the root CA cert you've generated):

curl https://mydomain:6006/

The test runner fails when trying to run tests:

$ yarn run test-storybook
yarn run v1.22.17
$ test-storybook --url "${STORYBOOK_URL:-https://mydomain.com:6006}"
[test-storybook] It seems that your Storybook instance is not running at: https://mydomain.com:6006/. Are you sure it's running?

If you're not running Storybook on the default 6006 port or want to run the tests against any custom URL, you can pass the --url flag like so:

yarn test-storybook --url http://localhost:9009

More info at https://github.com/storybookjs/test-runner#getting-started
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Expected behavior

The CURL should respond with HTML (as a sanity check) and the test runner should be able to run the tests.

Environment

• OS: Debian
• Node.js version: v16.13.0
• NPM version: 8.1.3
• Browser (if applicable): Chromium

[IanVS]

Thanks to your excellent reproduction instructions above, I was able to dig in a bit and I think I found a solution. It turns out it wasn't playwright causing the error, but node itself.

FetchError: request to https://mydomain.com:6006/ failed, reason: self signed certificate in certificate chain
    at ClientRequest.<anonymous> (/Users/ianvs/code/experiments/https-test-runner/node_modules/node-fetch/lib/index.js:1491:11)
    at ClientRequest.emit (node:events:394:28)
    at TLSSocket.socketErrorListener (node:_http_client:447:9)
    at TLSSocket.emit (node:events:394:28)
    at emitErrorNT (node:internal/streams/destroy:193:8)
    at emitErrorCloseNT (node:internal/streams/destroy:158:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  type: 'system',
  errno: 'SELF_SIGNED_CERT_IN_CHAIN',
  code: 'SELF_SIGNED_CERT_IN_CHAIN'
}

From the small bit of research I did, it seems like the best solution (based on https://stackoverflow.com/a/45088585/1435658) is to run:

NODE_TLS_REJECT_UNAUTHORIZED=0 npm test // <- or whatever script you have set up to run the test-runner

This keeps tls protection enabled in general, and disables it only for that one command. If anyone knows of a better solution, I'd love to hear it.

[KrofDrakula]

That sounded promising, but as I tried this, it doesn't work, same error. This makes sense, since I've added the generated root CA certificate to my system's trusted certificate store, but the browsers and Node don't automatically pick them up when initialising.

I've gotten Node to correctly pick up the trusted root CAs by passing in --use-openssl-ca as an argument, but the Playwright browser requests fail, as expected:

node --use-openssl-ca ./node_modules/.bin/test-storybook --url https://mydomain.com:6006

I would really like to find a way to pass on the trusted CAs on to the rest of the browsers, but I would settle for a fallback that ignores SSL errors. It seems that the test runner should expose a setting to disable that:

• https://github.com/microsoft/playwright/issues/4785

We need to forward the IgnoreHTTPSErrors in the BrowserNewContextOptions constructor, but I couldn't find the injection point in this codebase for it. Also, since this tool doesn't use @playwright/test, these docs are a bit moot:

https://playwright.dev/docs/api/class-testoptions#test-options-ignore-https-errors

I couldn't find any related function in the current Playwright preset, though, so maybe migrating to the recommended test runner would be better?

[IanVS]

Strange, I wonder why that worked for me, then. All I did was import the CA into my mac keychain and set it to "always trust", then the tests ran fine. But, if you think you need to set ignoreHttpsErrors, I think you can use the contextOptions shown here: https://github.com/playwright-community/jest-playwright#options. You may need to eject the test-runner to gain access to that config though.

[KrofDrakula]

I'm trying to run this on Debian where according to some sources Playwright-spawned browsers have their own certificate store separate from the OS', including Node, hence the use of --use-openssl-ca. Maybe the behaviour is different on Mac? Haven't tried that yet, as it's not my primary dev environment.