nx

nx copied to clipboard

Current Behavior

Nx local cache invalidation seems to be not working. If I change the content of a file listed on the inputs of the task and run the task again, I get an erroneous cache hit and the output of the task doesn't change.

Note that I have copied this repro from a closed issue (#18289), and fixing the error in output path in that reproduction does not prevent the cache miss, as demonstrated by the linked reproduction repo here.

Also here is a screenshot of it failing to rebuild after a change.

"useDaemonProcess": false fixes this for me, so I will be keeping that setting permanently from now.

Expected Behavior

A cache miss would be expected.

GitHub Repo

https://github.com/dsschneidermann/nx-cache-invalidation-bug-repro

Steps to Reproduce

1. Clone the repository
2. Run npm install
3. Run npm run test
4. Change the contents of the string inside the console.log in packages/test/src/index.js
5. Run npm run test
6. Compare the outputted string with the expected

Nx Report

>  NX   Report complete - copy this into the issue template

   Node   : 18.17.1
   OS     : linux-x64
   npm    : 9.6.7
   
   nx         : 16.7.4
   @nrwl/tao  : 16.7.4
   nx-cloud   : 16.1.1

Failure Logs

No response

Package Manager Version

No response

Operating System

• [ ] macOS
• [X] Linux
• [ ] Windows
• [ ] Other (Please specify)

Additional Information

No response

Still an issue on nx 16.8.1 / node 18.17.1 / npm 10.0.0 - also on Linux. This is reproducible even with small monorepos. Thankfully, adding"useDaemonProcess": false to the task runner options or setting the environment variable NX_DAEMON=false mitigate the issue for the time being.

Wow!! I finally found the issue, thanks @dsschneidermann. The same thing happens to me. I use runtime inputs to invalidate some caches, and after a few updates, it stopped working, making our dev environment quite unstable. By running with env NX_DAEMON=false, the cache gets invalidated again after repeated attempts. Awesome! 🙏

Is the fix on the team's radar?

Ran into the same issue today when setting up nx for the first time

My changes to the Nx config would never work driving me insane as I poured over the Nx documentation trying to figure out what I was doing wrong. However, I every time I rage quit and came back to it, it would suddenly work again

After a lot of debugging, I now finally realize the issue is caused by the Daemon who fails to reload any changes to my files. The reason rage-quitting causes it to work again is because the Daemons restarts after 3hrs of inactivity so when you come back to the task things magically work

You can fix this by adding "useDaemonProcess": false to your nx.json

"tasksRunnerOptions": {
    "default": {
      "runner": "nx/tasks-runners/default",
      "options": {
        "useDaemonProcess": false,
      }
    }
  },

Here's a repo with slightly different issue that seems related https://github.com/dmitry-stepanenko/nx-named-inputs-invalid-cache . Check the README for reproduction steps

I'm declaring a namedInput { "runtime": "node log.js" } and its being ignored by the Nx Daemon.

@dsschneidermann I tried running your repo in a Ubuntu install, and I couldn't reproduce your issue:

Do things start working properly if you do npx nx reset?

@dmitry-stepanenko we know about your issue with regards to the runtime input and envs. As a workaround for you, can you try using the env input directly instead of running it through the runtime input?

If that's not possible (because of a more complex node script), then I have the fix ready for that.

Thanks @Cammisuli, that's what I actually did. I created that reproduction with an intent to inform the team about something that seemed like a significant bug when I encountered it.

There was a conversation with @AgentEnder in the nx champions chat, quoting part of his response here for anyone else who faces similar behavior now

The main cause of the issue that you are seeing, is that when the daemon is spawned (and while it is alive) it has its own set of environment variables. They are static, and not updated based on the client env. So when you run a command with an inline environment variable (e.g. MY_VAR=123), that environment variable is set on the client side but the daemon is likely already alive, and thus doesn't see it. For env input's we have a workaround in place that fixes this. For runtime inputs its not as simple to workaround. That being said, the daemon is disabled in most cases on CI, unless its explicitly forced to be alive.

the fix for the runtime inputs using the env is here: 19422

@dsschneidermann I tried running your repo in a Ubuntu install, and I couldn't reproduce your issue:

Do things start working properly if you do npx nx reset?

@Cammisuli Thanks for looking into this. I suspect it has something to do with the file monitoring of the nx daemon and the way that Gitpod is running their virtual OS. You can replicate it with (a free account on) Gitpod: https://gitpod.io/#https://github.com/dsschneidermann/nx-cache-invalidation-bug-repro

I am not really aware of the underlying complexities here, but it seems clear that the watching fails to get notified. I am aware of plenty of other tools that are able to correctly monitor files in this specific virtual environment (eg. nodemon comes to mind), so the problem cannot be entirely placed on Gitpods shoulders. I suspect that the issue can also be reproduced locally in a docker-based environment - maybe relating somehow to host mounts and so on. It's certainly annoying that dev environments can vary on such a basic premise, and the fact that the Nx daemon is a default "on" can make it very difficult for some users to get started.

To answer your question, npx nx reset helps only once and then behaves the same again, so it doesn't provide any workaround.

sorry for the delay @dsschneidermann, but you're right, something doesnt work well on gitpod.. 🤔

I'm going to have to investigate this further.

I am running the latest version of nx 17.0.1 (node v18.18.2) on ubuntu 22.04 and have the same problem

We are seeing the same behavior on using NFS on macOS. Local filesystems work fine, but when we use networked shares on Mac, the watcher in the daemon never triggers. Other applications on Mac that are listening to file change work OK. And the same share on Linux works OK.

Stripped of actual paths, this is what nfsstat -vm says for one our our mount points:

/System/Volumes/path/to/mount/point from nfsserver:/exportpath
  -- Original mount options:
     General mount flags: 0x10500010 nodev,automounted,noatime,nobrowse
     NFS parameters: proto=tcp,noconn,locallocks,nordirplus,timeo=600,retrans=2
     File system locations:
       /exportpath @ nfsserver (dotquad)
  -- Current mount parameters:
     General mount flags: 0x14500010 nodev,automounted,noatime,nobrowse,multilabel
     NFS parameters: vers=3,proto=tcp,port=2049,nomntudp,hard,nointr,noresvport,negnamecache,callumnt,locallocks,quota,rsize=32768,wsize=32768,readahead=16,dsize=32768,nordirplus,nodumbtimer,timeo=600,maxgroups=16,acregmin=5,acregmax=60,acdirmin=5,acdirmax=60,nomutejukebox,nonfc,sec=sys,readlink_nocache=0
     File system locations:
       /exportpath @ nfsserver (dotquad)
     fh 20 01000600f77929ad00349f0c0000000000000000
     Current location: 0x1 0 0 0: 
       /exportpath @ nfsserver (dotquad)
     Status flags: 0x0

Ran into the same issue today when setting up nx for the first time

My changes to the Nx config would never work driving me insane as I poured over the Nx documentation trying to figure out what I was doing wrong. However, I every time I rage quit and came back to it, it would suddenly work again

After a lot of debugging, I now finally realize the issue is caused by the Daemon who fails to reload any changes to my files. The reason rage-quitting causes it to work again is because the Daemons restarts after 3hrs of inactivity so when you come back to the task things magically work

You can fix this by adding "useDaemonProcess": false to your nx.json

"tasksRunnerOptions": {
    "default": {
      "runner": "nx/tasks-runners/default",
      "options": {
        "useDaemonProcess": false,
      }
    }
  },

This fixed it for us also. We are using WSL. It won't recalculate, which causes really confusing issues where nx thinks that a projects hash hasn't changed but it has.

Same issue, disabled the daemon for now.

WSL / Ubuntu 22.04 / Nx 17.2.6

+1 WSL2 / Ubuntu 22.04.3 / Nx 17.2.8

"tasksRunnerOptions": {
    "default": {
      "runner": "nx/tasks-runners/default",
      "options": {
        "useDaemonProcess": false
      }
    }
  }

Disabling the daemon isn't helping.

I'm experiencing the same issue; the build cache gets used regardless of code changes. Turning cache off under "targetDefaults" > "build" works too:

"targetDefaults": {
    "build": {
      "cache": false,
      "dependsOn": ["^build"],
      "inputs": ["production", "^production"],
    },

But isn't build caching a core feature of NX? How could maintainers leave a bug like this for so long?

hey all, I'm using this to track the original issue of the daemon not working properly on gitpod. This is known, and Nx doesn't work well in that environment.

For everything else, can you make a new issue with clear steps, OSes used, environments that are interesting (ie, we test thoroughly on local file systems, what is different about your setup?) so that I can try my best to reproduce it?

This issue has been automatically marked as stale because it hasn't had any activity for 6 months. Many things may have changed within this time. The issue may have already been fixed or it may not be relevant anymore. If at this point, this is still an issue, please respond with updated information. It will be closed in 21 days if no further activity occurs. Thanks for being a part of the Nx community! 🙏