nx icon indicating copy to clipboard operation
nx copied to clipboard

Published 20 hours ago • verified publisher icon nrwl

Error reading nxdeps.json

Open yharaskrik opened this issue 2 years ago • 13 comments

Current Behavior

While running some commands in parallel we receive the following error:

Error reading '/home/circleci/project/repo/node_modules/.cache/nx/nxdeps.json'. Continue the process without the cache.
Error: ValueExpected in /home/circleci/project/repo/node_modules/.cache/nx/nxdeps.json at position 0
    at parseJson (/home/circleci/project/repo/node_modules/@nrwl/tao/src/utils/json.js:30:15)
    at readJsonFile (/home/circleci/project/repo/node_modules/@nrwl/tao/src/utils/fileutils.js:21:37)
    at readCache (/home/circleci/project/repo/node_modules/@nrwl/workspace/src/core/nx-deps/nx-deps-cache.js:41:46)
    at Object.readCachedProjectGraph (/home/circleci/project/repo/node_modules/@nrwl/workspace/src/core/project-graph/project-graph.js:19:61)
    at jestExecutor2 (/home/circleci/project/repo/node_modules/@*******org/nx-jest/src/executors/jest/jest.impl.js:107:46)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
[readCachedProjectGraph] ERROR: No cached ProjectGraph is available.

If you are leveraging \`readCachedProjectGraph()\` directly then you will need to refactor your usage to first ensure that
the ProjectGraph is created by calling \`await createProjectGraphAsync()\` somewhere before attempting to read the data.

(Note: we are using a jest executor that wraps the Nrwl Nx Jest executor to modify the jest config, doesn't do anything with Nx itself, can find the code here, https://github.com/trellisorg/platform/tree/master/packages/nx-jest, it shouldn't have anything to do with it.)

We have 3 e2e test suites (that actually just use jest) for our API, each of them has a dependsOn target:

"dependsOn": [
  { "target": "build", "projects": "dependencies" }
]

So that all dependencies of these e2e test suites (which are just node projects in the apps folder) are built (or pulled from cache) prior to running the e2e tests.

Then we run

yarn nx affected --target feature-e2e

and receive the error (not every time though)

Expected Behavior

Should not error and running commands in parallel should work fine.

Steps to Reproduce

This issue may not be prioritized if details are not provided to help us reproduce the issue.

Failure Logs

Error reading '/home/circleci/project/repo/node_modules/.cache/nx/nxdeps.json'. Continue the process without the cache.
Error: ValueExpected in /home/circleci/project/repo/node_modules/.cache/nx/nxdeps.json at position 0
    at parseJson (/home/circleci/project/repo/node_modules/@nrwl/tao/src/utils/json.js:30:15)
    at readJsonFile (/home/circleci/project/repo/node_modules/@nrwl/tao/src/utils/fileutils.js:21:37)
    at readCache (/home/circleci/project/repo/node_modules/@nrwl/workspace/src/core/nx-deps/nx-deps-cache.js:41:46)
    at Object.readCachedProjectGraph (/home/circleci/project/repo/node_modules/@nrwl/workspace/src/core/project-graph/project-graph.js:19:61)
    at jestExecutor2 (/home/circleci/project/repo/node_modules/@*******org/nx-jest/src/executors/jest/jest.impl.js:107:46)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
[readCachedProjectGraph] ERROR: No cached ProjectGraph is available.

If you are leveraging \`readCachedProjectGraph()\` directly then you will need to refactor your usage to first ensure that
the ProjectGraph is created by calling \`await createProjectGraphAsync()\` somewhere before attempting to read the data

Environment

 >  NX   Report complete - copy this into the issue template

   Node : 12.22.7
   OS   : darwin x64
   yarn : 1.22.17
   
   nx : 13.8.3
   @nrwl/angular : 13.8.3
   @nrwl/cli : 13.8.3
   @nrwl/cypress : 13.8.3
   @nrwl/detox : undefined
   @nrwl/devkit : 13.8.3
   @nrwl/eslint-plugin-nx : 13.8.3
   @nrwl/express : 13.8.3
   @nrwl/jest : 13.8.3
   @nrwl/js : 13.8.3
   @nrwl/linter : 13.8.3
   @nrwl/nest : 13.8.3
   @nrwl/next : undefined
   @nrwl/node : 13.8.3
   @nrwl/nx-cloud : 13.1.5
   @nrwl/react : undefined
   @nrwl/react-native : undefined
   @nrwl/schematics : undefined
   @nrwl/storybook : 13.8.3
   @nrwl/tao : 13.8.3
   @nrwl/web : 13.8.3
   @nrwl/workspace : 13.8.3
   typescript : 4.5.5
   rxjs : 7.4.0

yharaskrik avatar Feb 28 '22 18:02 yharaskrik

So it turns out this was because of the wrapper creating the project graph and then reading it right after. But then entirely unrelated I ran into it again (this time with tao)

Error reading '/home/circleci/project/repo/node_modules/.cache/nx/nxdeps.json'. Continue the process without the cache.
Error: ValueExpected in /home/circleci/project/repo/node_modules/.cache/nx/nxdeps.json at position 0
    at parseJson (/home/circleci/project/repo/node_modules/@nrwl/tao/src/utils/json.js:30:15)
    at readJsonFile (/home/circleci/project/repo/node_modules/@nrwl/tao/src/utils/fileutils.js:21:37)
    at readCache (/home/circleci/project/repo/node_modules/@nrwl/workspace/src/core/nx-deps/nx-deps-cache.js:41:46)
    at readCachedProjectGraph (/home/circleci/project/repo/node_modules/@nrwl/workspace/src/core/project-graph/project-graph.js:19:61)
    at /home/circleci/project/repo/node_modules/@nrwl/angular/src/executors/package/package.impl.js:41:146
    at Generator.next (<anonymous>)
    at resume (/home/circleci/project/repo/node_modules/@nrwl/angular/node_modules/tslib/tslib.js:225:48)
    at /home/circleci/project/repo/node_modules/@nrwl/angular/node_modules/tslib/tslib.js:224:125
    at new Promise (<anonymous>)
    at Object.i.<computed> [as next] (/home/circleci/project/repo/node_modules/@nrwl/angular/node_modules/tslib/tslib.js:224:67)
    at /home/circleci/project/repo/node_modules/@nrwl/tao/src/commands/run.js:101:35
    at Generator.next (<anonymous>)
    at /home/circleci/project/repo/node_modules/@nrwl/tao/node_modules/tslib/tslib.js:117:75
    at new Promise (<anonymous>)
    at __awaiter (/home/circleci/project/repo/node_modules/@nrwl/tao/node_modules/tslib/tslib.js:113:16)
    at iteratorToProcessStatusCode (/home/circleci/project/repo/node_modules/@nrwl/tao/src/commands/run.js:87:34)
[readCachedProjectGraph] ERROR: No cached ProjectGraph is available.

If you are leveraging \`readCachedProjectGraph()\` directly then you will need to refactor your usage to first ensure that
the ProjectGraph is created by calling \`await createProjectGraphAsync()\` somewhere before attempting to read the data.

If you encounter this error as part of running standard \`nx\` commands then please open an issue on https://github.com/nrwl/nx

yharaskrik avatar Mar 11 '22 22:03 yharaskrik

Also observing this. It's likely caused by concurrent reads/writes to nxdeps.json (while at position 0 seems to indicate that the file is always empty, we've sometimes observed it being non-zero, which is consistent with the process reading the file in the middle of a write operation executed by another process).

Logging changes to nxdeps.json during certain nx commands with https://github.com/inotify-tools/inotify-tools shows the following:

# For `npm run nx build <project>`
# This looks fine - create the file and _then_ read it
> inotifywait --monitor --recursive .nx
.nx/ CREATE nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ CLOSE_WRITE,CLOSE nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ ACCESS nxdeps.json
.nx/ CLOSE_NOWRITE,CLOSE nxdeps.json

# For `parallel 'npm run nx build' ::: <project_A> <project_B> <project_C> <project_D>` (basically 4 NX builds running in parallel)
# This doesn't look fine, reads and writes are interlaced
> inotifywait --monitor --recursive .nx
.nx/ CREATE nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ CLOSE_WRITE,CLOSE nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ CLOSE_WRITE,CLOSE nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ CLOSE_WRITE,CLOSE nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ CLOSE_WRITE,CLOSE nxdeps.json

If this is indeed the root cause possible solutions include:

  • Use something like https://github.com/npm/write-file-atomic to make sure that writes are atomic
  • Use a filesystem lock to serialize the operations
  • Add retry attempts to the read/write operation

sorin-davidoi avatar Mar 14 '22 13:03 sorin-davidoi

Can someone on the NX team weigh in on this (e.g. what solution would you merge if someone worked on it)?

sorin-davidoi avatar Mar 16 '22 09:03 sorin-davidoi

Bump, we currently migrating our Angular application to NX libraries on version 13.9.6. We have 1000+ modules and at this moment we migrated 100 of them and we had the same problem. It is a blocker for our firm because a lot of developers have this problem now. We found only one workaround, build every failed library separately

B1Z1 avatar Apr 06 '22 13:04 B1Z1

The problem is still exits in version 14.4.3

tamascsaba avatar Jul 20 '22 09:07 tamascsaba

The problem is still exits in version 14.5.1

erdemgonul avatar Aug 01 '22 13:08 erdemgonul

One workaround would be to disable the dependency graph cache completely by setting the NX_CACHE_PROJECT_GRAPH environment variable to false.

That being said, this looks like a deeper problem with concurrent runs while using a shared cache directory. If the NX team doesn't assure other concurrent problems can't happen in the future, it will never be safe for us developers to expect concurrent runs to work in future versions.

augustobmoura avatar Aug 01 '22 14:08 augustobmoura

Initially setting NX_CACHE_PROJECT_GRAPH just made the error happen on any nx command in the CI instead of just on concurrent calls.

So I instead set NX_DAEMON to false and that greatly helped (fixed it on all my primary code quality test) I wonder why this wasn't being caught already, the code seems to disable the domain for CI task already. I did run into issues when using https://www.npmjs.com/package/ngx-deploy-npm where it still gave the error.

wSedlacek avatar Aug 15 '22 11:08 wSedlacek

I noticed that the writeCache() is getting called over and over again just writing the same data that is already there every time the graph is read. Adding a simple check to see if the data it is writing is already what is there should reduce the chances that it is in the middle of writing when reading starts.

wSedlacek avatar Aug 15 '22 22:08 wSedlacek

Is this not solved by https://github.com/nrwl/nx/pull/10693 ? This will allow you to separate the shared chached artefacts from the nxdeps.json which we want not to share

jonhamm avatar Sep 08 '22 09:09 jonhamm

Is this not solved by #10693 ? This will allow you to separate the shared chached artefacts from the nxdeps.json which we want not to share

I don't believe so. As stated above, the issues with this seems to be unsafe interleaved writes to a (single) nxdeps.json file. Changing the file location alone isn't sufficient to address the issue.

kindrebelG avatar Sep 08 '22 16:09 kindrebelG

I don't believe so. As stated above, the issues with this seems to be unsafe interleaved writes to a (single) nxdeps.json file. Changing the file location alone isn't sufficient to address the issue.

But if you set NX_PROJECT_GRAPH_CACHE_DIRECTORY=.nx-project-cache then it will be local to each repo and thus not shared among multiple repos/build-nodes/worktrees?

jonhamm avatar Sep 08 '22 18:09 jonhamm

We still have it with v14.7.6 on our CI pipeline, even with NX_DAEMON set to false.

I haven't tried the NX_PROJECT_GRAPH_CACHE_DIRECTORY option, as I don't really understand how it could help for this issue, since it saves by default the nxdeps.json file under the node_modules. Our CI pipeline runs nx commands simultaneously (in parallel) within the same repo, other repos have their own node_modules folder. Setting it to another folder within the workspace will still result in the same issue I think?

> nx run jobs-e2e:lint

Error reading '/home/jenkins/workspace/Webpages/CubeMobileApps/node_modules/.cache/nx/nxdeps.json'. Continue the process without the cache.

Error: UnexpectedEndOfString in /home/jenkins/workspace/Webpages/CubeMobileApps/node_modules/.cache/nx/nxdeps.json at position 655323
    at parseJson (/home/jenkins/workspace/Webpages/CubeMobileApps/node_modules/nx/src/utils/json.js:23:15)
    at readJsonFile (/home/jenkins/workspace/Webpages/CubeMobileApps/node_modules/nx/src/utils/fileutils.js:23:37)
    at readCache (/home/jenkins/workspace/Webpages/CubeMobileApps/node_modules/nx/src/project-graph/nx-deps-cache.js:41:49)
    at readCachedProjectGraph (/home/jenkins/workspace/Webpages/CubeMobileApps/node_modules/nx/src/project-graph/project-graph.js:18:61)
    at run (/home/jenkins/workspace/Webpages/CubeMobileApps/node_modules/nx/src/command-line/run.js:167:69)
    at requireCli (/home/jenkins/workspace/Webpages/CubeMobileApps/node_modules/nx/bin/run-executor.js:20:23)
    at Object.<anonymous> (/home/jenkins/workspace/Webpages/CubeMobileApps/node_modules/nx/bin/run-executor.js:14:1)
    at Module._compile (node:internal/modules/cjs/loader:1126:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
    at Module.load (node:internal/modules/cjs/loader:1004:32)

Could not find 'nx' module in this workspace. Error: [readCachedProjectGraph] ERROR: No cached ProjectGraph is available.

If you are leveraging \`readCachedProjectGraph()\` directly then you will need to refactor your usage to first ensure that
the ProjectGraph is created by calling \`await createProjectGraphAsync()\` somewhere before attempting to read the data.

As workaround we read out this error message and run it again. Bit hacky, but it works...

Any update on this? For us this is a critical issue, I'm a bit surprised to see this still open after about half a year...

wimme avatar Sep 20 '22 08:09 wimme

I don't believe so. As stated above, the issues with this seems to be unsafe interleaved writes to a (single) nxdeps.json file. Changing the file location alone isn't sufficient to address the issue.

But if you set NX_PROJECT_GRAPH_CACHE_DIRECTORY=.nx-project-cache then it will be local to each repo and thus not shared among multiple repos/build-nodes/worktrees?

On our end at least, we are running into the issue on a single NX driven monorepository in a CI environment that is not sharing volumes with other build agents. We use yarn3, and node modules are hoisted to root of repo. So there is only a single node_modules/.cache/nx/nxdeps.json file. As @wimme states, I don't see how NX_PROJECT_GRAPH_CACHE_DIRECTORY could solve this.

kindrebelG avatar Sep 22 '22 20:09 kindrebelG

I don't believe so. As stated above, the issues with this seems to be unsafe interleaved writes to a (single) nxdeps.json file. Changing the file location alone isn't sufficient to address the issue.

But if you set NX_PROJECT_GRAPH_CACHE_DIRECTORY=.nx-project-cache then it will be local to each repo and thus not shared among multiple repos/build-nodes/worktrees?

On our end at least, we are running into the issue on a single NX driven monorepository in a CI environment that is not sharing volumes with other build agents. We use yarn3, and node modules are hoisted to root of repo. So there is only a single node_modules/.cache/nx/nxdeps.json file. As @wimme states, I don't see how NX_PROJECT_GRAPH_CACHE_DIRECTORY could solve this.

We have exactly the same issue, running standalone agents running builds in parallel on jenkins and randomly crashing with Error reading nxdeps.json

maximegasque avatar Oct 04 '22 15:10 maximegasque

And now I have seen this issue multiple times too on a Jenkins build node where we run only one branch build at any point in time, so I totally agree that NX_PROJECT_GRAPH_CACHE_DIRECTORY will not solve the problem. It is quite frustrating that this is not solved yet

Error: ValueExpected in /home/jenkins/.nxcache/nxdeps.json at position 1765376
    at parseJson (/home/jenkins/jenkins-workspace/workspace/build-all-PRs/src.1/node_modules/nx/src/utils/json.js:23:15)
    at readJsonFile (/home/jenkins/jenkins-workspace/workspace/build-all-PRs/src.1/node_modules/nx/src/utils/fileutils.js:23:37)
    at readCache (/home/jenkins/jenkins-workspace/workspace/build-all-PRs/src.1/node_modules/nx/src/project-graph/nx-deps-cache.js:41:49)
    at readCachedProjectGraph (/home/jenkins/jenkins-workspace/workspace/build-all-PRs/src.1/node_modules/nx/src/project-graph/project-graph.js:19:61)
    at run (/home/jenkins/jenkins-workspace/workspace/build-all-PRs/src.1/node_modules/nx/src/command-line/run.js:167:69)
    at requireCli (/home/jenkins/jenkins-workspace/workspace/build-all-PRs/src.1/node_modules/nx/bin/run-executor.js:20:23)
    at Object.<anonymous> (/home/jenkins/jenkins-workspace/workspace/build-all-PRs/src.1/node_modules/nx/bin/run-executor.js:14:1)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
Could not find 'nx' module in this workspace. Error: [readCachedProjectGraph] ERROR: No cached ProjectGraph is available.

If you are leveraging \`readCachedProjectGraph()\` directly then you will need to refactor your usage to first ensure that
the ProjectGraph is created by calling \`await createProjectGraphAsync()\` somewhere before attempting to read the data.

If you encounter this error as part of running standard \`nx\` commands then please open an issue on https://github.com/nrwl/nx
    at readCachedProjectGraph (/home/jenkins/jenkins-workspace/workspace/build-all-PRs/src.1/node_modules/nx/src/project-graph/project-graph.js:28:15)
    at run (/home/jenkins/jenkins-workspace/workspace...

jonhamm avatar Oct 10 '22 09:10 jonhamm

We are also seeing this error in Nx 14.4.3. Luckily, we have Nx Cloud so we added retry logic around our Nx commands to account for this, but it's less than ideal

koliveira15 avatar Oct 19 '22 16:10 koliveira15

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.

github-actions[bot] avatar Mar 21 '23 12:03 github-actions[bot]