Turbo randomly gets stuck on build process

[faustbrian]

What version of Turborepo are you using?

1.2.6

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Mac

Describe the Bug

I have a relatively large private repository (65 packages and growing) in which I'm attempting to use Turbo but running into an issue with builds that I don't encounter with any other tool. When I run turbo run build it'll start to build and then randomly get stuck and do nothing, e.g. go halfway through building and then just stop and sit there. This doesn't happen deterministically on the same package every time, which could be an indicator for a circular dependency but there are none.

This issue also doesn't exists if I run the workspace build via pnpm or use https://github.com/folke/ultra-runner with ultra -r build instead of turbo run build. The setup isn't anything crazy, it's just the stock pnpm setup from https://github.com/vercel/turborepo/tree/main/examples/with-pnpm with 65 packages inside the packages directory.

Is there anything I could or should look at to debug this? Loving how blazing fast Turbo is but this build issue prevents me from fully utilising it.

Expected Behavior

Finish the build process without any issues like pnpm and ultra do.

To Reproduce

It's a relatively large private repository so not sure what the best way to provide reproduction steps would be. The configuration is fairly simple:

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": [
        "^build"
      ],
      "inputs": [
        "src/**/*.ts",
        "package.json"
      ],
      "outputs": [
        "distribution/**"
      ]
    }
  }
}

[nathanhammond]

Hey @faustbrian! Thanks for the report! Even though it doesn't stop at the same package every time, does turbo stop every time?

Can you also share your pnpm-workspace.yaml to help improve my chances of reproduction?

[faustbrian]

The pnpm-workspace.yaml file is just the basic template and turbo does indeed stop every single time, just not deterministically at the same point, which makes it a bit of a pain to debug this.

packages:
  - "packages/*"

[nathanhammond]

Last question: is it possibly timing out after spending a certain amount of time running the process? Or is the duration of the execution before stopping also inconsistent?

(Just trying to build a hypothesis to check with a reproduction.)

[faustbrian]

The execution time seems fairly consistent, give or take 1 second. I'm not sure if it's relevant but the project has various native module dependencies like https://github.com/bcoin-org/bcrypto and also various WASM dependencies, not sure if this could cause any issues for Turbo.

[wouter-leistra]

Not sure if it is related but we have also a Monorepo with 35 or so packages and when running yarn turbo run test with or without filter it sometimes gets stuck, when I cancel the process in the terminal manually I get to see that all tests were passed (which is correct) but turborepo never picked up on the end signal by jest it seems... We use 1.2.8. Jest 27.

[mikestopcontinues]

I'm having similar issues. I was noticing it with my test command, so I tried switching from jest to vitest. No help.

Then I found that adding additional steps to the pipeline (via dependsOn) causes the process to hang at a roughly different place. I tried --concurrency=1 and --concurrency=100%. It seems nondeterministic, save that it tends to hang around a similar spot given a constant concurrency.

If you press ctrl+c when it's hanging early in the pipeline, it will end the inner process, allowing turbo to proceed. However, if it's hanging at the end of the pipeline, ctrl+c will exit the pipeline without the summary ever getting printed. (Perhaps the command is getting captured by turbo in this case.)

I'm using yarn 3.2.0 and the most recent version of turborepo, but I've been having this problem for a while, with many different versions. It's only become pressing now because I'm trying to use turbo in CI/CD rather that yarn workspaces.

[Edu-mt]

I experienced the same problem in a fairly large repo (80 packages). The build process gets stuck and memory usage grows until the system becomes unresponsive. This was not happening on version 1.2.5 but started to happen with version 1.2.6.

[mikestopcontinues]

Reverting to 1.2.5 doesn't solve it for me.

[weyert]

I have similar issues were it works fine on my Mac but fails on Gitlab CI. Downgrading to 1.2.5 appears to improve things.

[gsoltis]

Do dry runs (--dry) produce any issues, or is it only when actually executing tasks?

It's possible that there is some child-process management bug.

[mikestopcontinues]

@gsoltis Dry run works for me. Consistently fails otherwise.

[camsjams]

Check that there are no cyclic dependencies, or worse, self dependencies. In an unfortunate find and replace incident, a package relied on itself in dependencies, and this caused a silent failure in turborepo with the random hanging effect described here.

If your project is broken on 1.2.5 and 1.2.6, an invalid dependency will still cause it to hang.

[wiredmatt]

I'm experiencing the same but running yarn test, my build runs fine though.

"scripts": {
    "test": "react-scripts test",
  },

The test:

import { render, screen } from '@testing-library/react'

test('renders react component', () => {
  render(<div>hello react</div>)
  const divElement = screen.getByText(/hello react/i)
  expect(divElement).toBeInTheDocument()
})

The output:

admin-dashboard:test: $ react-scripts test
admin-dashboard:test: PASS src/App.test.tsx
admin-dashboard:test:   ✓ renders react component (19 ms)
admin-dashboard:test: 
admin-dashboard:test: Test Suites: 1 passed, 1 total
admin-dashboard:test: Tests:       1 passed, 1 total
admin-dashboard:test: Snapshots:   0 total
admin-dashboard:test: Time:        0.613 s, estimated 1 s
admin-dashboard:test: Ran all test suites related to changed files.
admin-dashboard:test: 

Looks like that's a problem with react-scripts test not with turborepo, mb!

"test": "react-scripts test --watchAll=false"

[ghost]

I've experienced similar issues in CI environments (GitHub Actions, Amplify). I was running tsc build for each one out of ~70 packages in a monorepo and the build would fail randomly at more or less the same spot but without any hint at what went wrong.

Setting --concurrency=100% fixed it. Since concurrency is set to 10 by default, could it be Turbo was trying to allocate more resources than the environment allowed?

[jaunkst]

@system32uwu I am experiencing the same with just plain jest.

@angi/ldm:test: cache miss, executing 7e9c114985b5d77a
@angi/ldm:test: PASS test/commands/system/test.test.ts
@angi/ldm:test:   system:test
@angi/ldm:test:     ✓ with no args or flags (469 ms)
@angi/ldm:test:
@angi/ldm:test: Test Suites: 1 passed, 1 total
@angi/ldm:test: Tests:       1 passed, 1 total
@angi/ldm:test: Snapshots:   0 total
@angi/ldm:test: Time:        3.27 s, estimated 4 s
@angi/ldm:test: Ran all test suites.

Jest has moved on, and it's still hanging here. I can do jest && echo 'Done' and I will see the echo after jest reports its done running all test suites. I've even wrapped the command in a bash script and did an exit 0 after and still nothing. I'm really not sure how to resolve this. I've seen other strange behaviors when modifying the test to just a simple case I this behavior goes away. Either way, the node process even exits that's running jest and it just sits here.

[wiredmatt]

@jaunkst Does anything else run after that jest test?

[jaunkst]

@system32uwu yes and no.. Turbo hangs out here, but whatever task is defined in the package.json for test will allow for a following process to be executed after the unit.

[jaunkst]

wrapping my call in a wrapping process seams to fix this issue for me. However it leaves a running zombie node process. 🤷‍♂️ scripts/run-test.js

import { execSync } from "child_process";

execSync("yarn jest", { stdio: "inherit" })

package.json

"scripts": {
    "test": "node ./scripts/run-tests.js"
}

Ive logged the node process PIDS created by jest, and this parent wrapping process and they are all exiting. The PID that remains seems to be created at the very end when the task is complete.

Forcing this PID to debug mode reveals it is some sort of RPC worker sync-rpc/lib/worker.js

[jaunkst]

I found my issue. Looks like it's not related to turborepo, but a bad actor in a dependency that I can replace.

With that said if your task seems to be hung, check your processes for node instances that are spawning, and not being cleaned up. This was the issue in my case, and I did the following to find out what was going on.

Allow the node instance to be inspected

kill -s SIGUSR1 <pid of zombie node process>

Then go to chrome and open chrome://inspect

[mattoni]

What was the dependency? I'm having the same issue. How did you find what instances were spawning?

[jaunkst]

@mattoni look at your monitor node processes with ps or activity monitor. I'm my case I noticed a node instance spawned during my unit tests and was not cleaned up. I then inspected it with the above approach to find the a library using child_process.spawnSync that wasn't exiting.

[tjx666]

Same here:

[cimchd]

I had the same problem. It was caused by a wrong dependency entry in the package.json of a package where I had the package itself as a dependency:

// package.json
{
  "name": "@pkg/my-package",
  "dependencies": {
    "@pkg/my-package": "*"
  }
}

The package that caused the hang did not produced any output in the console. You can check this with the packages in scope listed by turbo at the beginning of the command.

Maybe this helps someone.

[GiancarlosIO]

My case is a little different It gets stuck with the label cache-hit 🤔 . If I force to build (TURBO_FORCE=true) the build finishes without any error 🤔 .