Daemon file watcher slow to start causing stale cache and project graph in large repos using pnpm workspace

[ajkarell]

Current Behavior

When starting the Nx daemon in a large project (in my case over 150 libraries) that uses PNPM workspaces, the file watching behavior doesn't activate until multiple minutes have passed. I presume this occurs due to the file watcher or hasher going through every library, each of which containing symlinks to each other inside node_modules. Having deleted most of the library-specific node_modules (specifically the nested workspace libraries), the file watching begins much quicker. Once it begins, one can run pnpm install to restore the library-specific node_modules and utilize the daemon normally.

This causes Nx cache to not work correctly (because file hashes aren't being recomputed). Probably also causes the project graph to become invalid if dependencies are modified.

Expected Behavior

The daemon file watcher should activate quickly after daemon startup, preventing stale cache and project graph.

GitHub Repo

No response

Steps to Reproduce

Hard to reproduce, requires large monorepo using PNPM workspaces with lots of libraries and dependencies. This problem is prevalent on Linux. My colleague using macOS on Apple Silicon doesn't seem to experience this.

Nx Report

Node           : 22.21.1
OS             : linux-x64
Native Target  : x86_64-linux
pnpm           : 10.22.0

nx                     : 22.2.0
@nx/js                 : 22.2.0
@nx/jest               : 22.2.0
@nx/eslint             : 22.2.0
@nx/workspace          : 22.2.0
@nx/cypress            : 22.2.0
@nx/devkit             : 22.2.0
@nx/esbuild            : 22.2.0
@nx/eslint-plugin      : 22.2.0
@nx/module-federation  : 22.2.0
@nx/node               : 22.2.0
@nx/playwright         : 22.2.0
@nx/react              : 22.2.0
@nx/rollup             : 22.2.0
@nx/storybook          : 22.2.0
@nx/vite               : 22.2.0
@nx/vitest             : 22.2.0
@nx/web                : 22.2.0
@nx/docker             : 22.2.0
typescript             : 5.9.3

---------------------------------------
Nx key licensed packages

@nx/azure-cache  : 5.0.0

---------------------------------------
Registered Plugins:
@nx/eslint/plugin
@nx/vite/plugin
@nx/js/typescript
@nx/vitest
@nx/js/typescript
@nx/storybook/plugin
@nx/docker
---------------------------------------
Community plugins:
@nx/azure-cache : 5.0.0
---------------------------------------

Failure Logs

Package Manager Version

pnpm 10.22.0

Operating System

• [ ] macOS
• [x] Linux
• [ ] Windows
• [ ] Other (Please specify)

Additional Information

I've checked that all node_modules directories are in .gitignore and .nxignore. Modifying a file inside any node_modules directory doesn't result in any daemon log output, contrary to any other non-ignored file. I'm also not hitting any inotify watcher limits, and there are no error logs overall. The same issue occurs with Nx 22.1.3.

My current workaround is this:

1. Delete all workspace dependencies from subdirectory node_modules: find . -type d -path './*/**node_modules/@my-workspace' -not -path './node_modules**' -exec rm -rf {} +
2. Start the Nx daemon: pnpm nx
3. Wait and test that file watching is working, e.g. watch logs with tail -F .nx/workspace-data/d/daemon.log and modify a file, you should see something like [WATCHER]: libs/my-lib/src/index.ts was modified
4. Once the watcher works, restore all dependencies with pnpm install

Example script (replace @acme with correct name):

#!/usr/bin/env bash

# Workaround for https://github.com/nrwl/nx/issues/33781

# Reset the daemon
echo "Resetting the daemon..."
pnpm nx reset

# Remove library specific node_modules
echo "Removing library specific node_modules..."
find . -type d -path './*/**node_modules/@acme' -not -path './node_modules**' -exec rm -rf {} +

# Start the daemon
echo "Starting the daemon..."
pnpm nx daemon

# Wait for file watcher to run
echo "Waiting for file watcher to run..."
while true; do
  echo "Waiting for file watcher to run..." > tmp_wait_for_daemon
  sleep 0.5
  # Check that daemon log contains 'Processing file changes in outputs'
  if grep -q 'Processing file changes in outputs' .nx/workspace-data/d/daemon.log; then
    break
  fi
  # Check that daemon log contains 'tmp_wait_for_daemon'
  if grep -q 'tmp_wait_for_daemon' .nx/workspace-data/d/daemon.log; then
    break
  fi
done

rm tmp_wait_for_daemon

# Restore dependencies
echo "Restoring dependencies..."
pnpm install