Occasional flake when setting up node

[mockdeep]

trafficstars

Orb version:

5.0.0

What happened:

We occasionally see flaky builds where node or NPM fails to install. For example, here's the output from one recent failure:

failure output

#!/bin/bash -eo pipefail
# Only install nvm if it's not already installed
if command -v nvm &> /dev/null; then
    echo "nvm is already installed. Skipping nvm install.";
else
    curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash;
    
    echo 'export NVM_DIR="$HOME/.nvm"' >> "$BASH_ENV";
    echo '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" --no-use' >> "$BASH_ENV";
    
    # shellcheck source=/dev/null
    source "$BASH_ENV";
fi

# See: https://github.com/nvm-sh/nvm#usage
if [ "$NODE_PARAM_VERSION" = "latest" ]; then
    # When no version is specified we default to the latest version of Node
    NODE_ORB_INSTALL_VERSION=$(nvm ls-remote | tail -n1 | grep -Eo 'v[0-9]+\.[0-9]+\.[0-9]+')
    nvm install "$NODE_ORB_INSTALL_VERSION" # aka nvm install node. We're being explicit here.
    nvm alias default "$NODE_ORB_INSTALL_VERSION"
    elif [ -n "$NODE_PARAM_VERSION" ]; then
    nvm install "$NODE_PARAM_VERSION"
    nvm alias default "$NODE_PARAM_VERSION"
    elif [ -f ".nvmrc" ]; then
    NVMRC_SPECIFIED_VERSION=$(<.nvmrc)
    nvm install "$NVMRC_SPECIFIED_VERSION"
    nvm alias default "$NVMRC_SPECIFIED_VERSION"
else
    nvm install --lts
    nvm alias default lts/*
fi

echo 'nvm use default &>/dev/null' >> "$BASH_ENV"

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13527  100 13527    0     0  1016k      0 --:--:-- --:--:-- --:--:-- 1016k
=> Downloading nvm from git to '/home/circleci/.nvm'
=> Cloning into '/home/circleci/.nvm'...
ssh: connect to host github.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Failed to clone nvm repo. Please report this!

Exited with code exit status 2

CircleCI received exit code 2

Expected behavior:

Node should reliably install in the container.

[Jaryt]

Hi @mockdeep ! Thanks for submitting this issue. Can you inform us on how often this occurs? I believe this might be an issue with the install script provided by nvm, or with github being flaky on clone connections. While searching around for issues like this, I kept stumbling across posts where people are changing the port from ssh to ssl: https://gist.github.com/kcc0/d88150ace547976cfcccd578f30c816a

However, I think that might be a solution to when the 22 port is blocked by a firewall, which doesn't apply here.

[mockdeep]

Hi @Jaryt, thanks for the quick response. We see this issue a handful of times each week. I think it's probably our largest source of flake in our test suite at the moment.

Out of curiosity, I was looking at how asdf installs node and it looks like they use node-build. I've got no idea how that is different from nvm, but maybe worth looking into?

Also, should the installation be cached so that it doesn't need to be reinstalled each time? I wonder if that might reduce the flake some, too.

[Jaryt]

I'd have to investigate the caching of .nvm further to ensure that it's stable. But since you are encountering this, I would encourage you to experiment with caching /home/circleci/.nvm and letting us know how you fair! (I will experiment with this as well, once I get the chance)

[mockdeep]

This is failing pretty hard for us right now. I haven't had a chance to try the caching, yet, but we're seeing a ton of failures with exit status 18. It's hard to tell anything else about why it is failing from the output:

output

#!/bin/bash -eo pipefail
if [[ $EUID == 0 ]]; then export SUDO=""; else export SUDO="sudo"; fi

# FUNCTIONS
get_yarn_version () {
    if [[ "$NODE_PARAM_YARN_VERSION" == "" ]]; then
    YARN_ORB_VERSION=$(curl -Ls -o /dev/null -w "%{url_effective}" \
        "https://github.com/yarnpkg/yarn/releases/latest" | sed 's:.*/::' | cut -d 'v' -f 2 | cut -d 'v' -f 2)
    echo "Latest version of Yarn is $YARN_ORB_VERSION"
    else
    YARN_ORB_VERSION="$NODE_PARAM_YARN_VERSION"

    echo "Selected version of Yarn is $YARN_ORB_VERSION"
    fi
}

installation_check () {
    echo "Checking if YARN is already installed..."
    if command -v yarn > /dev/null 2>&1; then
    if yarn --version | grep "$YARN_ORB_VERSION" > /dev/null 2>&1; then
        echo "Yarn $YARN_ORB_VERSION is already installed"
        exit 0
    else
        echo "A different version of Yarn is installed ($(yarn --version)); removing it"

        if uname -a | grep Darwin > /dev/null 2>&1; then
        brew uninstall yarn > /dev/null 2>&1
        elif grep Alpine /etc/issue > /dev/null 2>&1; then
        apk del yarn > /dev/null 2>&1
        elif grep Debian /etc/issue > /dev/null 2>&1; then
        $SUDO apt-get remove yarn > /dev/null 2>&1 && \
            $SUDO apt-get purge yarn > /dev/null 2>&1
        elif grep Ubuntu /etc/issue > /dev/null 2>&1; then
        $SUDO apt-get remove yarn > /dev/null 2>&1 && \
            $SUDO apt-get purge yarn > /dev/null 2>&1
        elif command -v yum > /dev/null 2>&1; then
        yum remove yarn > /dev/null 2>&1
        fi

        $SUDO rm -rf "$HOME/.yarn" > /dev/null 2>&1
        $SUDO rm -f /usr/local/bin/yarn /usr/local/bin/yarnpkg > /dev/null 2>&1
    fi
    fi
}

get_yarn_version
installation_check

# install yarn
echo "Installing YARN v$YARN_ORB_VERSION"
curl -L -o yarn.tar.gz --silent "https://yarnpkg.com/downloads/$YARN_ORB_VERSION/yarn-v$YARN_ORB_VERSION.tar.gz"

$SUDO tar -xzf yarn.tar.gz && rm yarn.tar.gz

$SUDO mkdir -p /opt/yarn
$SUDO mv yarn-v"${YARN_ORB_VERSION}"/* /opt/yarn

$SUDO rm -rf "yarn-v${YARN_ORB_VERSION}"

$SUDO chmod 777 "/opt/yarn"

$SUDO ln -s /opt/yarn/bin/yarn /usr/local/bin/yarn
$SUDO ln -s /opt/yarn/bin/yarnpkg /usr/local/bin/yarnpkg
$SUDO ln -s /opt/yarn/bin/yarn.js /usr/local/bin/yarn.js

$SUDO mkdir -p ~/.config

if uname -a | grep Darwin; then
    $SUDO chown -R "$USER:$GROUP" ~/.config
    $SUDO chown -R "$USER:$GROUP" /opt/yarn
else
    $SUDO chown -R "$(whoami):$(whoami)" /opt/yarn
    $SUDO chown -R "$(whoami):$(whoami)" ~/.config
fi

# test/verify version
echo "Verifying YARN install"
if yarn --version | grep "$YARN_ORB_VERSION" > /dev/null 2>&1; then
    echo "Success! Yarn $(yarn --version) has been installed to $(which yarn)"
else
    echo "Something went wrong; the specified version of Yarn could not be installed"
    exit 1
fi

Latest version of Yarn is 1.22.17
Checking if YARN is already installed...
A different version of Yarn is installed (1.22.5); removing it
Installing YARN v1.22.17

Exited with code exit status 18

CircleCI received exit code 18

[Jaryt]

Moving to backlog as we will be escalating this to our images team.

[marboledacci]

I see this issue is been open for a while, and many things have changed in the images, @mockdeep please confirm if this is still an issue or we can close.

[mockdeep]

@marboledacci we actually set up our own custom Dockerfile with node built in, so we haven't had to deal with this issue in a while. I'm fine closing this.