Persistent tasks in watch mode don't wait for first run of their dependencies to finish

[OliverJAsh]

Verify canary release

• [X] I verified that the issue exists in the latest Turborepo canary release.

Link to code that reproduces this issue

https://github.com/OliverJAsh/turbo-watch-deps-of-persistent

What package manager are you using / does the bug impact?

Yarn v2/v3/v4 (node_modules linker only)

What operating system are you using?

Mac

Which canary version will you have in your reproduction?

2.0.6-canary.0

Describe the Bug

Persistent tasks in watch mode don't wait for first run of their dependencies to finish.

• Package b depends on package a.
• Both packages have a build task.

In non-watch / run mode, build a correctly runs before build b:

$ TURBO_UI=false yarn run turbo run build --filter b
• Packages in scope: b
• Running build in 1 packages
• Remote caching disabled
a:build: cache bypass, force executing 7c09bd93554e1d1c
a:build: A
b:build: cache bypass, force executing 7f551e27d3b8aa66
b:build: B

 Tasks:    2 successful, 2 total
Cached:    0 cached, 2 total
  Time:    1.516s

However, in watch mode, Turbo runs both builds in parallel. Build b does not wait for build a to finish even though it's a dependency:

$ TURBO_UI=false yarn run turbo watch build --filter b
• Packages in scope: b
• Running build in 1 packages
• Remote caching disabled
• Packages in scope: b
• Running build in 1 packages
• Remote caching disabled
a:build: cache bypass, force executing 7c09bd93554e1d1c
b:build: cache bypass, force executing 25c5580a26314aac
b:build: B
a:build: A

Expected Behavior

Persistent tasks in watch mode should wait for the first run of their dependencies to finish, otherwise they may fail.

To Reproduce

1. Clone https://github.com/OliverJAsh/turbo-watch-deps-of-persistent.
2. Run yarn.
3. Run TURBO_UI=false yarn run turbo watch build --filter b.

Additional context

I believe the idea is that persistent tasks should self-heal: https://github.com/vercel/turbo/issues/8164#issuecomment-2122878984

However, this is inefficient and can lead to confusing error messages.

In our case we're using webpack watch and rebuilding is CPU intensive, so it would be better if webpack waited for its dependencies to finish building before starting.

Furthermore, our webpack watch task can only self-heal after the build has started—it can't self-heal if the configuration itself is missing dependencies.

We only expect the order to be respected for the first run, not subsequent to that.

[NicholasLYang]

Hi, thanks for the issue. We currently don't have the capacity to work on this currently. If you want, you're welcome to open a PR for it and I can provide guidance.

[Patrick-Ullrich]

@NicholasLYang IMHO this feels like a huge blocker of turborepo adoption. Given it isn't a priority I feel like I am missing something. As someone trying to use turbo watch since it showed up in canary I don't quite understand what it is currently solving without this. Is there an example repository somewhere that would show how someone can leverage turbo watch?

If i am understanding correctly, the usual set up is:

• apps with persistent dev tasks (next, vite, etc.)
• packages with a build step

Whenever a package changes, the dependent apps need to know about it. Without it, one would have to restart turbo dev on every change.

I tried various work arounds:

• make dev dependent on dev and add a bundler to every package that supports some sort of watch mode, but you can't make persistent tasks dependent on other persistent tasks
• remove persistent, and remove all hot reloads, hasn't been very succesful

The best solution for now has been using packages without a build step and just export the ts files directly as much as possible and let the host apps bundler bundle the package instead.

Appreciate any guidance / insight!

[NicholasLYang]

Hi @Patrick-Ullrich, sorry for the late reply! Could you try using the new interruptible flag? This sort of does what you want, where we now include persistent tasks in the regular task graph if they're marked as interruptible. The trade-off is that we also restart them when we encounter a change. Depending on your use case, that could be what you want.

[Patrick-Ullrich]

@NicholasLYang That sounds like it would do the trick! Part of 2.2.3?

Should it work like this:

  "dev": {
      "dependsOn": [
        "^build"
      ],
      "cache": false,
      "persistent": true,
      "interruptible": true
    },

Also happy to take this conversation somewhere else if we are getting too off topic!

[KajSzy]

I'm trying to cover very similar case where packages are build while main app uses webpack dev server. During watch mode packages are properly built and re-build but as soon as they are done webpack is being shut down and restarted. Tried using interruptible flag but it nothing has changed

[gregnr]

@KajSzy I experienced something similar to this and it turned out it was a bug in the latest v2.2.3 release where persistent tasks were accidently stopped in watch mode: https://github.com/vercel/turborepo/pull/9330. Fixed in v2.2.4-canary.2+

[KajSzy]

Yes, with 2.3 release it was fixed

[thejerrybao]

Is there a way to configure it so that only the non-persistent tasks are executed after the first run?

I have a configuration like so:

    "build": {
      "dependsOn": [
        "^build"
      ],
      "outputs": [
        "dist/**"
      ]
    },
    "dev": {
      "dependsOn": ["^build"],
      "cache": false,
      "persistent": true,
      "interruptible": true,
    },

and while this does wait for the first run of dependencies to finish, subsequent changes to packages also runs dev again. dev for me is a Nest dev server that doesn't need to be rerun.

[theogravity]

I'm facing the same situation in [email protected] (also checked with [email protected]), except I have next.js in dev mode running along with an API server that auto-reloads on its own when something has changed.

This is without the interruptible flag - next.js / the API server runs at the same time as the dependencies build as opposed to the dependency build finishing first THEN starting the services.

    "build:dev": {
      "dependsOn": [
        "^build:dev"
      ],
      "cache": false
    },
    "dev": {
      "dependsOn": [
        "^build:dev"
      ],
      "cache": false,
      "persistent": true
    },

(I expect build:dev to complete first before running the items in dev)

Like the OP, turbo dev is fine, but turbo watch dev is bugged.

[theogravity]

I did an awful hack that involved the use of the wait-on package:

• I created two packages (I call them lockfile packages), each representing my next.js and my API service, respectively
• In each package, I added the following command to the package.json: "build:dev": "rm -rf .lockfile",
  • The packages have the dependencies that needed to be built listed as devDependencies before that service can start
• I installed wait-on as a devDep in my next.js and API service
• I augmented the dev script on both next.js and API service with touch ../../packages/<service-name>-lock/.lockfile && wait-on -r ../../packages/<service-name>-lock/.lockfile && <service start watch command>

So what happens when I run the turbo watch dev command:

• Because of this bug, the services will run concurrently to the dependency build. However, when the service runs, they will create .lockfile and then wait for that .lockfile to be removed before actually starting the server
• The packages will build, and because the lockfile packages have those packages as a dependency, it will only remove the lockfile once those dependent packages have been built
• Because the lockfile is now removed, then the services will start with the correctly built dependencies

[dylanjmcdonald]

There still seems to be something going on with watch. I'm following the kitchen sink api example and either turbo watch or tsup --watch isn't detecting changes.

[dmstoykov]

@dylanjmcdonald In the context of the kitchen sink example, if you change the code inside of @repo/logger the api tsup --watch won't detect the change as its not aware when one of its dependencies change.

This led me to then make the following changes:

1. Change the dev script in the root package.json to turbo watch dev
2. Delete the dev script inside of the packages/logger and packages/ui because we are using the turbo watch command now
3. Make the dev task inside of api interruptible so that the api server is restarted when its dependencies change

This leaves us with the following setup:

1. Apps inside the apps/ directory have dev scripts that are both persistent but only apps/api is interruptible
2. Packages inside the packages/ directory have a build script and rely on turbo watch to be rebuilt

However, with such setup (which i believe is an extremely common one in monorepos) there is a bug currently ([email protected]).

Repro steps:

1. Update the code inside packages/ui package
2. packages/ui gets rebuilt (as it should)
3. The interruptible api dev task gets terminated and isn't restarted.

I believe this bug is documented here => https://github.com/vercel/turborepo/issues/9421

I haven't been able to find a good workaround for this issue.

UPDATE: A neat workaround that I found was to mark all persistent tasks as non-interruptible and use the watch command from tsx to monitor and restart those tasks. tsx watch (v4.19.2) also detects changes inside internal package dependencies of the watched package.

[iobuhov]

I stack with this issue on [email protected], adding interruptible to dev task fixed the issue.

{
"tasks": {
    "build": {
      "inputs": ["$TURBO_DEFAULT$", ".env*"],
      "dependsOn": ["^build"],
      "outputs": ["dist/**", ".next/**", "!.next/cache/**"]
    },
    "dev": {
      "cache": false,
      "persistent": true,
      "dependsOn": ["^build"],
      "interruptible": true
    },
    "clean": {
      "cache": false
    }
  }
}

[johnpyp]

I think I'm hitting a similar issue here...

I have a non interruptible dev and an interruptible dev. Running either of them works fine, but when running both, one of them will just mysteriously hang and never actually execute!

Like others have said, it works perfectly fine using run instead of watch.

---

EDIT: My bug was in fact https://github.com/vercel/turborepo/issues/9421, like mentioned above. Interruptible tasks getting killed.

[theogravity]

@dylanjmcdonald In the context of the kitchen sink example, if you change the code inside of @repo/logger the api tsup --watch won't detect the change as its not aware when one of its dependencies change.

This led me to then make the following changes:

1. Change the dev script in the root package.json to turbo watch dev
2. Delete the dev script inside of the packages/logger and packages/ui because we are using the turbo watch command now
3. Make the dev task inside of api interruptible so that the api server is restarted when its dependencies change

This leaves us with the following setup:

1. Apps inside the apps/ directory have dev scripts that are both persistent but only apps/api is interruptible
2. Packages inside the packages/ directory have a build script and rely on turbo watch to be rebuilt

However, with such setup (which i believe is an extremely common one in monorepos) there is a bug currently ([email protected]).

Repro steps:

1. Update the code inside packages/ui package
2. packages/ui gets rebuilt (as it should)
3. The interruptible api dev task gets terminated and isn't restarted.

I believe this bug is documented here => #9421

I haven't been able to find a good workaround for this issue.

UPDATE: A neat workaround that I found was to mark all persistent tasks as non-interruptible and use the watch command from tsx to monitor and restart those tasks. tsx watch (v4.19.2) also detects changes inside internal package dependencies of the watched package.

Can you share your turbo config?

[vitormarkis]

Running on this bug too

[fbartho]

Using turbo 2.4.4, setting interruptible: true on the dev task config in the root turbo.json still produced flaky dev execution on my system.

I've moved interruptible: true to the package that kept failing to execute, ~and if that's reliable I plan to distribute to that setting to all the leaf packages with dev commands.~

UPDATE: That was not reliable, I'm still looking for a solution here.

[alexandreh92]

In my case (using "turbo": "2.5.2"), a setting like this works out of the box when running pnpm turbo watch dev, it waits for the dependencies's build and then start the command. However, I could not prevent the restart of this command when I change a single piece of code (in apps/sandbox). Even by adding a static file in inputs/outputs config.

// turbo.json in apps/sandbox - parent #build is packages/ui
{
  ...
  "tasks": {
    "dev": {
      "dependsOn": ["^build"],
      "persistent": true,
      "interruptible": true,
      "cache": false
    }
  }
}

Not sure if I did something wrong, but as the doc says interruptible is doing more than one job:

• Sets task to "wait" the dependency graph;
• Restart the task if changes detected.

This is very confusing and maybe if you don't pass "interruptible" it's assuming that your hot-reload should handle the errors that occur?

Anyways, I think a property is missing when persistent is true that can control the Sets task to "wait" the dependency graph; step, or ignore the "restart/reload" step.