CREATE_FAILED Custom::RFDK_X509Generator openssl: command not found

[aleksander-mendoza]

I get the following error

7:15:02 AM | CREATE_FAILED        | Custom::RFDK_X509Generator                  | RenderQueueRootCA4708D079
Received response status [FAILED] from custom resource. Message returned: Command failed: openssl req -x509 -passout env:CERT_PASSPHRASE -newkey
rsa:2048 -days 1095 -extensions v3_ca -keyout /tmp/tmp.metU8N/key -out /tmp/tmp.metU8N/crt -subj /CN=RenderQueueRootCA/O=AWS/OU=Thinkbox
/bin/sh: openssl: command not found

Error: Command failed: openssl req -x509 -passout env:CERT_PASSPHRASE -newkey rsa:2048 -days 1095 -extensions v3_ca -keyout /tmp/tmp.metU8N/key -
out /tmp/tmp.metU8N/crt -subj /CN=RenderQueueRootCA/O=AWS/OU=Thinkbox
/bin/sh: openssl: command not found

at ChildProcess.exithandler (node:child_process:402:12)
at ChildProcess.emit (node:events:513:28)
at ChildProcess.emit (node:domain:489:12)
at maybeClose (node:internal/child_process:1100:16)
at Socket.<anonymous> (node:internal/child_process:458:11)
at Socket.emit (node:events:513:28)
at Socket.emit (node:domain:489:12)
at Pipe.<anonymous> (node:net:301:12) (RequestId: 7a1fa905-f9e4-4643-9db6-9d250c8ec59c)

Reproduction Steps

This is my CDK code

import * as cdk from 'aws-cdk-lib';
import { aws_s3 as s3, 
  aws_lambda as lambda, 
  aws_apigateway as apigateway, 
  aws_ecr_assets as ecr_assets, 
  aws_ecr as ecr,
  aws_ec2 as ec2,
  aws_efs as efs } from 'aws-cdk-lib';
import * as ecrdeploy from 'cdk-ecr-deployment';
import {deadline as deadline} from 'aws-rfdk';
import * as rdfk from 'aws-rfdk';
import { Construct } from 'constructs';
import * as path from 'path';
import * as fs from 'fs';

function build_thinkbox(scope: Construct){
  const vpc = new ec2.Vpc(scope, 'Vpc', { maxAzs: 2 });
  const version = new deadline.VersionQuery(scope, 'Version', {
    version: '10.2.0',
  });
  let imgs = new deadline.ThinkboxDockerImages(scope,'Thinkbox Images', {
    version:version,
    userAwsCustomerAgreementAndIpLicenseAcceptance: deadline.AwsCustomerAgreementAndIpLicenseAcceptance.USER_ACCEPTS_AWS_CUSTOMER_AGREEMENT_AND_IP_LICENSE
  });
  const repo = new deadline.Repository(scope, 'Repository', {
    vpc:vpc,
    version:version,
    // Allow resources to be deleted when we delete the sample
    removalPolicy: {
      database: cdk.RemovalPolicy.DESTROY,
      filesystem: cdk.RemovalPolicy.DESTROY
    },
  });
  const renderQueue = new deadline.RenderQueue(scope, 'RenderQueue', {
    vpc:vpc,
    version:version,
    images:imgs.forRenderQueue(),
    repository:repo,
    // Allow the load-balancer to be deleted when we delete the sample
    deletionProtection: false,
  });
}

export class AppStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    build_thinkbox(this);
  }
}

this is my package.json

{
  "name": "app",
  "version": "0.1.0",
  "bin": {
    "app": "bin/app.js"
  },
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "@types/jest": "^29.5.4",
    "@types/node": "20.5.9",
    "aws-cdk": "2.70.0",
    "jest": "^29.6.4",
    "ts-jest": "^29.1.1",
    "ts-node": "^10.9.1",
    "typescript": "~5.2.2"
  },
  "dependencies": {
    "aws-cdk-lib": "2.70.0",
    "aws-rfdk": "1.2.0",
    "cdk-ecr-deployment": "^2.5.30",
    "constructs": "^10.0.0",
    "source-map-support": "^0.5.21"
  }
}

I'm running inside docker

FROM ubuntu:20.04

# Environment variables
ENV DEBIAN_FRONTEND noninteractive
ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8
# Install essentials
RUN apt-get update && apt-get install -y ca-certificates curl gnupg build-essential
# Keyring for node.js and npm
RUN mkdir -p /etc/apt/keyrings
RUN curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
RUN echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_20.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list
# Keyring for docker
RUN install -m 0755 -d /etc/apt/keyrings
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
RUN chmod a+r /etc/apt/keyrings/docker.gpg
RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | tee /etc/apt/sources.list.d/docker.list
# Install node.js , npm and docker
RUN apt-get update
RUN apt-get install -y nodejs docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Install rust and wasm-pack
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain nightly -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN echo $PATH
RUN cargo install wasm-pack
# Install cdk and typescript
RUN npm -g install typescript aws-cdk

which I invoke with the following parameters

docker run -it \
        -v "$PROJECT_ROOT:/mnt" \    # mount project files
        -v "$HOME/.aws:/root/.aws" \     # mount AWS profile (so that cdk command works)
        -v "//var/run/docker.sock:/var/run/docker.sock"  \   # mount docker socket (so that docker works)
        my_docker_tag

then inside docker I do

cd /mnt
cdk deploy

I also get an identical error when running on my windows host instead of inside docker. I have openssl installed and on PATH both inside docker

$ which openssl
/usr/bin/openssl

and on host machine

> gcm openssl

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     openssl.exe                                        1.1.1.13   C:\tools\miniconda3\Library\bin\openssl.exe

Error Log

This is the full error log

7:15:02 AM | CREATE_FAILED        | Custom::RFDK_X509Generator                  | RenderQueueRootCA4708D079
Received response status [FAILED] from custom resource. Message returned: Command failed: openssl req -x509 -passout env:CERT_PASSPHRASE -newkey
rsa:2048 -days 1095 -extensions v3_ca -keyout /tmp/tmp.metU8N/key -out /tmp/tmp.metU8N/crt -subj /CN=RenderQueueRootCA/O=AWS/OU=Thinkbox
/bin/sh: openssl: command not found

Error: Command failed: openssl req -x509 -passout env:CERT_PASSPHRASE -newkey rsa:2048 -days 1095 -extensions v3_ca -keyout /tmp/tmp.metU8N/key -
out /tmp/tmp.metU8N/crt -subj /CN=RenderQueueRootCA/O=AWS/OU=Thinkbox
/bin/sh: openssl: command not found

at ChildProcess.exithandler (node:child_process:402:12)
at ChildProcess.emit (node:events:513:28)
at ChildProcess.emit (node:domain:489:12)
at maybeClose (node:internal/child_process:1100:16)
at Socket.<anonymous> (node:internal/child_process:458:11)
at Socket.emit (node:events:513:28)
at Socket.emit (node:domain:489:12)
at Pipe.<anonymous> (node:net:301:12) (RequestId: 7a1fa905-f9e4-4643-9db6-9d250c8ec59c)


 ❌  AppStack failed: Error: The stack named AppStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: Command failed: openssl req -x509 -passout env:CERT_PASSPHRASE -newkey rsa:2048 -days 1095 -extensions v3_ca -keyout /tmp/tmp.metU8N/key -out /tmp/tmp.metU8N/crt -subj /CN=RenderQueueRootCA/O=AWS/OU=Thinkbox
/bin/sh: openssl: command not found

Error: Command failed: openssl req -x509 -passout env:CERT_PASSPHRASE -newkey rsa:2048 -days 1095 -extensions v3_ca -keyout /tmp/tmp.metU8N/key -out /tmp/tmp.metU8N/crt -subj /CN=RenderQueueRootCA/O=AWS/OU=Thinkbox
/bin/sh: openssl: command not found

    at ChildProcess.exithandler (node:child_process:402:12)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess.emit (node:domain:489:12)
    at maybeClose (node:internal/child_process:1100:16)
    at Socket.<anonymous> (node:internal/child_process:458:11)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at Pipe.<anonymous> (node:net:301:12) (RequestId: 7a1fa905-f9e4-4643-9db6-9d250c8ec59c)
    at FullCloudFormationDeployment.monitorDeployment (/usr/lib/node_modules/aws-cdk/lib/index.js:467:10232)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/usr/lib/node_modules/aws-cdk/lib/index.js:470:179911)
    at async /usr/lib/node_modules/aws-cdk/lib/index.js:470:163159

 ❌ Deployment failed: Error: The stack named AppStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: Command failed: openssl req -x509 -passout env:CERT_PASSPHRASE -newkey rsa:2048 -days 1095 -extensions v3_ca -keyout /tmp/tmp.metU8N/key -out /tmp/tmp.metU8N/crt -subj /CN=RenderQueueRootCA/O=AWS/OU=Thinkbox
/bin/sh: openssl: command not found

Error: Command failed: openssl req -x509 -passout env:CERT_PASSPHRASE -newkey rsa:2048 -days 1095 -extensions v3_ca -keyout /tmp/tmp.metU8N/key -out /tmp/tmp.metU8N/crt -subj /CN=RenderQueueRootCA/O=AWS/OU=Thinkbox
/bin/sh: openssl: command not found

    at ChildProcess.exithandler (node:child_process:402:12)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess.emit (node:domain:489:12)
    at maybeClose (node:internal/child_process:1100:16)
    at Socket.<anonymous> (node:internal/child_process:458:11)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at Pipe.<anonymous> (node:net:301:12) (RequestId: 7a1fa905-f9e4-4643-9db6-9d250c8ec59c)
    at FullCloudFormationDeployment.monitorDeployment (/usr/lib/node_modules/aws-cdk/lib/index.js:467:10232)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/usr/lib/node_modules/aws-cdk/lib/index.js:470:179911)
    at async /usr/lib/node_modules/aws-cdk/lib/index.js:470:163159

The stack named AppStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: Command failed: openssl req -x509 -passout env:CERT_PASSPHRASE -newkey rsa:2048 -days 1095 -extensions v3_ca -keyout /tmp/tmp.metU8N/key -out /tmp/tmp.metU8N/crt -subj /CN=RenderQueueRootCA/O=AWS/OU=Thinkbox
/bin/sh: openssl: command not found

Error: Command failed: openssl req -x509 -passout env:CERT_PASSPHRASE -newkey rsa:2048 -days 1095 -extensions v3_ca -keyout /tmp/tmp.metU8N/key -out /tmp/tmp.metU8N/crt -subj /CN=RenderQueueRootCA/O=AWS/OU=Thinkbox
/bin/sh: openssl: command not found

    at ChildProcess.exithandler (node:child_process:402:12)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess.emit (node:domain:489:12)
    at maybeClose (node:internal/child_process:1100:16)
    at Socket.<anonymous> (node:internal/child_process:458:11)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at Pipe.<anonymous> (node:net:301:12) (RequestId: 7a1fa905-f9e4-4643-9db6-9d250c8ec59c)

Environment

• CDK CLI Version : 2.98.0 (build b04f852)
• CDK Framework Version: 2.70.0
• RFDK Version: 1.2.0
• Deadline Version: none
• Node.js Version: v20.7.0
• OS : both ubuntu 20.04 and windows 10
• Language (Version): TypeScript (5.2.2)

Other

---

This is :bug: Bug Report

[ddneilson]

Thanks for the report, Aleksander! We'll dig in to it

[aleksander-mendoza]

I've been running this on eu-west-1 the whole time. I just tried to switch to us-east-1 and now everything works perfectly fine. So the issue is basically that something is wrong with those lambdas here on different regions.

https://github.com/aws/aws-rfdk/blob/mainline/packages/aws-rfdk/lib/lambdas/lambdaLayerVersionArns.ts