Large Monorepo OOM Errors with Turbosnap Enabled

[ChrisSargent]

Bug report

We are seeing OOM errors during the Chromatic build process if we have Turbosnap enabled. In particular this seems to happen if there's any package.json change causing a change to the yarn.lock file.

We have tried setting max-old-space pretty high but it still happens.

Critical Logs are below, nothing interesting in the stacktrace.

Found 99 manifest/lockfile pairs to diff

<--- Last few GCs --->

[4282:0x6cc28b0]   194031 ms: Scavenge (reduce) 3924.3 (4142.0) -> 3923.9 (4142.3) MB, 5.7 / 0.0 ms  (average mu = 0.144, current mu = 0.066) allocation failure; 
[4282:0x6cc28b0]   194042 ms: Scavenge (reduce) 3924.8 (4142.3) -> 3924.4 (4142.5) MB, 5.2 / 0.0 ms  (average mu = 0.144, current mu = 0.066) allocation failure; 
[4282:0x6cc28b0]   194051 ms: Scavenge (reduce) 3925.2 (4142.5) -> 3924.8 (4142.8) MB, 4.8 / 0.0 ms  (average mu = 0.144, current mu = 0.066) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Aborted (core dumped)

This started happening after this upgrade (and we also started getting errors about the traceChanged option seemingly changed from expand to expanded.

[cgbl-90]

hello @ChrisSargent have you tried the workarounds described here: Why is my build failing with an Out of memory error?

[ChrisSargent]

Hi @cgbl-90 - yes, we have 4gb of max old space size set in NODE_OPTIONS. We run in to the same issues locally on 16GB ram macbook pros.

FYI, we've also trying setting the max old space size much higher but it eventually dies.

[codykaup]

@ChrisSargent thanks for reporting this!

It appears to be coming from our lock file parsing lib and improving that is going to take quite a bit of effort (which we're discussing internally). For now though, I merged in #1140 which should help with preventing the OOM errors at the cost of running a full Chromatic build when dependencies change and the lock file is too large to parse.

Feel free to give v11.23.0/latest a try and let us know if that helps!

If you continue to see issues, try setting the environment variable MAX_LOCK_FILE_SIZE to a lower number (that value is in bytes) until it no longer throws that OOM error.

[ChrisSargent]

@codykaup - thanks for the response we can give it a try and report back. For reference our lock file is only 2.2MB. To work around in the last few months, we've just been skipping turbosnap if yarn.lock changed at all:

      - name: 'Setup: Check yarn.lock changes'
        run: |-
          UNCHANGED_YARN_LOCK=$(./scripts/cicd/checkfile-unchanged.sh yarn.lock)
          echo "UNCHANGED_YARN_LOCK=$UNCHANGED_YARN_LOCK" >> $GITHUB_ENV
      - name: 'Build & Deploy: Storybook'
        run: chromatic --only-changed=$UNCHANGED_YARN_LOCK
        env:
          CHROMATIC_PROJECT_TOKEN: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}

#!/bin/bash

# Check if a specified file has changed since the last commit
FILE_TO_CHECK=$1

if git diff --name-only HEAD^ HEAD | grep -q "$FILE_TO_CHECK"; then
  # File is not unchanged
  echo "false"
else
  # File is unchanged
  echo "true"
fi

[codykaup]

Well that's a bummer, we might have to lower our original value then. 😞

You (or anybody else that might come across this) should be able to set MAX_LOCK_FILE_SIZE to something like 1 to force a full rebuild on every lock file change if you don't want to use a custom workflow like this.

[awinograd]

Adding a data point, we just bumped chromatic from 11.5.1 to 11.25.1 with a yarn.lock of 1.2 MB and are seeing OOM on CI.

[ChrisSargent]

@codykaup - just to confirm, does this only bail out of Turbosnap if the lock file changed && it is > MAX_LOCK_FILE_SIZE? Or does it bail out on all runs, regardless of if the lock file actually changed?

[codykaup]

@codykaup - just to confirm, does this only bail out of Turbosnap if the lock file changed && it is > MAX_LOCK_FILE_SIZE? Or does it bail out on all runs, regardless of if the lock file actually changed?

@ChrisSargent great question! It will only bail out if there are lock file changes and it's larger than MAX_LOCK_FILE_SIZE.

For context, there's 2 parts to Turbosnap:

1. Gathering changed source files based on git history
2. Gathering changed files based on dependency changes

(2) Only runs if the git history says there was a lock file change. If there wasn't a lock file change, it's skipped entirely.

[ChrisSargent]

Confirming that setting MAX_LOCK_FILE_SIZE=1 is working for us and we are able to remove our custom code.

[codykaup]

@ChrisSargent @awinograd FYI we upgraded our lock file parsing algorithm in v11.27.0 which may improve the situation here. Feel free to give it a test if you'd like!