nx
nx copied to clipboard
nrwl
nx daemon dies due to javascript heap memory issue. (Watch error: Daemon closed the connection)
Current Behavior
The nx daemon dies due to a memory issue, causing the server to restart periodically.
Expected Behavior
nx daemon works without memory issues
GitHub Repo
No response
Steps to Reproduce
- Create an image based on docker node:20.11.
- Install [email protected] [email protected] in the image.
- Only apps/libs and etc configure file (nx.json, package.json) are mounted on the host.
- Set “nx serve app” as Docker Entrypoint and run the application with Docker.
- Docker dies periodically with the error “Watch error: Daemon closed the connection”.
Nx Report
root@dbaa33480eca:/backend# nx report
> NX Report complete - copy this into the issue template
Node : 20.11.1
OS : linux-arm64
pnpm : 8.15.1
nx (global) : 18.0.2
nx : 18.0.2
@nx/js : 18.0.2
@nx/jest : 18.0.2
@nx/eslint : 18.0.2
@nx/workspace : 18.0.2
@nx/eslint-plugin : 18.0.2
@nx/nest : 18.0.2
@nx/node : 18.0.2
@nx/web : 18.0.2
@nx/webpack : 18.0.2
typescript : 5.3.3
Failure Logs
Server Log
NX Daemon Logs
[NX Daemon Server] - 2024-02-23T07:23:28.585Z - [WATCHER]: 2 file(s) created or restored, 2 file(s) modified, 2 file(s) deleted
[NX Daemon Server] - 2024-02-23T07:23:28.586Z - Time taken for 'changed-projects' 0.13487500000337604ms
[NX Daemon Server] - 2024-02-23T07:23:28.639Z - [WATCHER]: .nx/cache/d/daemon.log was modified
[NX Daemon Server] - 2024-02-23T07:23:28.644Z - Time taken for 'changed-projects' 0.09233299999323208ms
[NX Daemon Server] - 2024-02-23T07:23:28.691Z - [WATCHER]: .nx/cache/d/daemon.log was modified
[NX Daemon Server] - 2024-02-23T07:23:28.691Z - Time taken for 'changed-projects' 0.04025000000547152ms
[NX Daemon Server] - 2024-02-23T07:23:28.743Z - [WATCHER]: .nx/cache/d/daemon.log was modified
[NX Daemon Server] - 2024-02-23T07:23:28.744Z - Time taken for 'changed-projects' 0.03812500000640284ms
[NX Daemon Server] - 2024-02-23T07:23:28.773Z - [REQUEST]: Updated workspace context based on watched changes, recomputing project graph...
[NX Daemon Server] - 2024-02-23T07:23:28.773Z - [REQUEST]: .nx/cache/d/daemon.log,.nx/cache/file-map.json,.nx/cache/project-graph.json,.nx/cache/jest.hash
[NX Daemon Server] - 2024-02-23T07:23:28.773Z - [REQUEST]: .nx/cache/file-map.json~991a0e34,.nx/cache/project-graph.json~991a0e34
<--- Last few GCs --->
[88:0xfe7ebf0] 94175 ms: Scavenge (reduce) 1967.2 (2083.5) -> 1966.3 (2083.5) MB, 2.12 / 0.00 ms (average mu = 0.104, current mu = 0.005) allocation failure;
[88:0xfe7ebf0] 94179 ms: Scavenge (reduce) 1967.2 (2083.5) -> 1966.5 (2083.7) MB, 2.19 / 0.00 ms (average mu = 0.104, current mu = 0.005) allocation failure;
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
1: 0xc8640c node::Abort() [/usr/local/bin/node]
2: 0xb755c8 void node::FPrintF<>(_IO_FILE*, char const*) [/usr/local/bin/node]
3: 0xe8450c v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/local/bin/node]
4: 0xe846dc v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/local/bin/node]
5: 0x108c3ec [/usr/local/bin/node]
6: 0x108c908 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/local/bin/node]
7: 0x10a20f0 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [/usr/local/bin/node]
8: 0x10a273c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
9: 0x107b62c v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0x107c3f0 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0x105b410 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/local/bin/node]
12: 0x104cc88 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [/usr/local/bin/node]
13: 0x104f44c v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [/usr/local/bin/node]
14: 0x1367988 v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::AllocationType) [/usr/local/bin/node]
15: 0x136a938 v8::internal::String::SlowEquals(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
16: 0x148e034 v8::internal::Runtime_StringEqual(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
17: 0x186da84 [/usr/local/bin/node]
[NX Daemon Server] - 2024-02-23T07:23:33.293Z - Started listening on: /tmp/c92b5041713b6818abf4/d.sock
[NX Daemon Server] - 2024-02-23T07:23:33.294Z - [WATCHER]: Subscribed to changes within: /backend (native)
[NX Daemon Server] - 2024-02-23T07:23:33.300Z - Established a connection. Number of open connections: 1
[NX Daemon Server] - 2024-02-23T07:23:33.300Z - Established a connection. Number of open connections: 2
[NX Daemon Server] - 2024-02-23T07:23:33.301Z - Closed a connection. Number of open connections: 1
[NX Daemon Server] - 2024-02-23T07:23:33.301Z - [REQUEST]: Client Request for Project Graph Received
[NX Daemon Server] - 2024-02-23T07:23:33.307Z - [REQUEST]: Updated workspace context based on watched changes, recomputing project graph...
[NX Daemon Server] - 2024-02-23T07:23:33.307Z - [REQUEST]:
[NX Daemon Server] - 2024-02-23T07:23:33.307Z - [REQUEST]:
[NX Daemon Server] - 2024-02-23T07:23:33.574Z - [REQUEST]: Responding to the client. project-graph
[NX Daemon Server] - 2024-02-23T07:23:33.575Z - Time taken for 'total for creating and serializing project graph' 272.310459ms
[NX Daemon Server] - 2024-02-23T07:23:33.575Z - Done responding to the client project-graph
[NX Daemon Server] - 2024-02-23T07:23:33.575Z - Handled REQUEST_PROJECT_GRAPH. Handling time: 273. Response time: 1.
[NX Daemon Server] - 2024-02-23T07:23:33.580Z - Established a connection. Number of open connections: 2
[NX Daemon Server] - 2024-02-23T07:23:33.819Z - [WATCHER]: 2 file(s) created or restored, 3 file(s) modified, 2 file(s) deleted
[NX Daemon Server] - 2024-02-23T07:23:33.820Z - Time taken for 'changed-projects' 0.1638749999999618ms
[NX Daemon Server] - 2024-02-23T07:23:33.872Z - [WATCHER]: .nx/cache/d/daemon.log was modified
[NX Daemon Server] - 2024-02-23T07:23:33.872Z - Time taken for 'changed-projects' 0.03954200000009678ms
[NX Daemon Server] - 2024-02-23T07:23:33.924Z - [REQUEST]: Updated workspace context based on watched changes, recomputing project graph...
[NX Daemon Server] - 2024-02-23T07:23:33.924Z - [REQUEST]: .nx/cache/file-map.json,.nx/cache/project-graph.json,.nx/cache/jest.hash,.nx/cache/nx_files.nxt,.nx/cache/d/daemon.log
[NX Daemon Server] - 2024-02-23T07:23:33.924Z - [REQUEST]: .nx/cache/project-graph.json~6efe5306,.nx/cache/file-map.json~6efe5306
[NX Daemon Server] - 2024-02-23T07:23:34.079Z - [WATCHER]: 63 file(s) created or restored, 4 file(s) modified, 2 file(s) deleted
[NX Daemon Server] - 2024-02-23T07:23:34.080Z - [WATCHER]: 1 file(s) created or restored, 2 file(s) modified, 0 file(s) deleted
[NX Daemon Server] - 2024-02-23T07:23:34.080Z - [WATCHER]: Processing file changes in outputs
[NX Daemon Server] - 2024-02-23T07:23:34.080Z - Time taken for 'hash changed files from watcher' 3.4336250000000064ms
[NX Daemon Server] - 2024-02-23T07:23:34.121Z - [WATCHER]: 2 file(s) created or restored, 2 file(s) modified, 2 file(s) deleted
[NX Daemon Server] - 2024-02-23T07:23:34.121Z - Time taken for 'changed-projects' 0.03195800000003146ms
[NX Daemon Server] - 2024-02-23T07:23:34.174Z - [WATCHER]: .nx/cache/d/daemon.log was modified
[NX Daemon Server] - 2024-02-23T07:23:34.174Z - Time taken for 'changed-projects' 0.027749999999969077ms
[NX Daemon Server] - 2024-02-23T07:23:34.182Z - [REQUEST]: Updated workspace context based on watched changes, recomputing project graph...
[NX Daemon Server] - 2024-02-23T07:23:34.182Z - [REQUEST]: dist/libs/datahub/src/rdb/repository/repository.provider.js.map,dist/libs/datahub/src/rdb/repository/repository.provider.js,dist/libs/util/src/promise/promise-lock.d.ts,dist/libs/datahub/src/rdb/entity/4x/move.entity.js.map,dist/libs/datahub/src/rdb/repository/4x/index.js.map,dist/libs/datahub/src/rdb/entity/4x/index.d.ts,dist/libs/util/src/promise/index.d.ts,dist/libs/datahub/src/rdb/entity/4x/post.entity.js,dist/libs/datahub/src/rdb/entity/4x/item.entity.js.map,dist/libs/datahub/src/rdb/repository/repository.error.js,dist/libs/datahub/src/rdb/entity/4x/index.js,dist/libs/datahub/src/rdb/entity/main.js.map,dist/libs/datahub/src/rdb/connection/connection.error.js.map,dist/libs/datahub/src/rdb/repository/abstractEntity.repository.js.map,dist/libs/datahub/src/rdb/repository/abstract.repository.js.map,dist/libs/datahub/src/rdb/entity/4x/user.entity.js,dist/libs/datahub/src/rdb/entity/4x/item.entity.d.ts,dist/libs/datahub/src/rdb/entity/4x/move.entity.js,dist/libs/datahub/src/rdb/repository/abstractEntity.repository.d.ts,dist/libs/datahub/src/rdb/entity/4x/post.entity.d.ts,dist/libs/util/src/promise/promise-lock.js,dist/libs/datahub/src/rdb/connection/data-source.module.js,dist/libs/datahub/src/rdb/connection/data-source.js.map,dist/libs/datahub/src/rdb/entity/main.d.ts,dist/libs/datahub/src/rdb/connection/data-source.js,dist/libs/datahub/src/rdb/connection/connection.js.map,dist/libs/datahub/src/rdb/connection/data-source.module.js.map,dist/libs/datahub/src/rdb/repository/4x/index.js,dist/libs/datahub/src/rdb/entity/4x/item.entity.js,dist/libs/datahub/src/rdb/entity/4x/post.entity.js.map,dist/libs/datahub/src/rdb/connection/connection.d.ts,dist/libs/datahub/src/rdb/index.d.ts,.nx/cache/file-map.json,dist/libs/util/README.md,dist/libs/datahub/src/rdb/index.js.map,.nx/cache/project-graph.json,dist/libs/datahub/src/rdb/repository/abstract.repository.js,dist/libs/datahub/src/rdb/entity/4x/index.js.map,dist/libs/datahub/src/rdb/entity/main.js,dist/libs/datahub/src/rdb/connection/connection.error.js,dist/libs/datahub/src/rdb/connection/connection.error.d.ts,dist/libs/datahub/src/rdb/repository/abstractEntity.repository.js,dist/libs/util/src/promise/index.js.map,dist/libs/util/package.json,dist/libs/datahub/src/rdb/repository/4x/item.repository.d.ts,dist/libs/datahub/src/rdb/entity/4x/move.entity.d.ts,dist/libs/datahub/src/rdb/repository/abstract.repository.d.ts,dist/libs/datahub/src/rdb/repository/4x/item.repository.js.map,dist/libs/datahub/README.md,dist/libs/datahub/src/rdb/connection/data-source.d.ts,dist/libs/datahub/src/rdb/repository/4x/index.d.ts,dist/libs/datahub/src/rdb/repository/repository.error.d.ts,dist/libs/datahub/src/rdb/repository/4x/item.repository.js,dist/libs/util/src/promise/index.js,dist/libs/datahub/src/rdb/repository/repository.provider.d.ts,dist/libs/datahub/src/rdb/entity/4x/user.entity.d.ts,dist/libs/util/src/promise/promise-lock.js.map,dist/libs/datahub/package.json,dist/libs/datahub/src/rdb/connection/data-source.module.d.ts,dist/libs/datahub/src/rdb/entity/4x/user.entity.js.map,dist/libs/datahub/src/rdb/index.js,dist/libs/datahub/src/rdb/repository/repository.error.js.map,dist/libs/datahub/src/rdb/connection/connection.js,.nx/cache/jest.hash,dist/apps/grpc-server/main.js.map,dist/apps/grpc-server/main.js,.nx/cache/d/daemon.log,.nx/cache/run.json
[NX Daemon Server] - 2024-02-23T07:23:34.183Z - [REQUEST]: .nx/cache/file-map.json~8faf8a0d,.nx/cache/project-graph.json~8faf8a0d,.nx/cache/file-map.json~3bdf05ca,.nx/cache/project-graph.json~3bdf05ca
[NX Daemon Server] - 2024-02-23T07:23:34.344Z - [WATCHER]: .nx/cache/d/daemon.log was modified
[NX Daemon Server] - 2024-02-23T07:23:34.345Z - Time taken for 'hash changed files from watcher' 1.7638749999999845ms
[NX Daemon Server] - 2024-02-23T07:23:34.387Z - [WATCHER]: 2 file(s) created or restored, 2 file(s) modified, 2 file(s) deleted
[NX Daemon Server] - 2024-02-23T07:23:34.387Z - Time taken for 'changed-projects' 0.06087499999989632ms
[NX Daemon Server] - 2024-02-23T07:23:34.440Z - [WATCHER]: .nx/cache/d/daemon.log was modified
[NX Daemon Server] - 2024-02-23T07:23:34.441Z - Time taken for 'changed-projects' 0.027833000000100583ms
[NX Daemon Server] - 2024-02-23T07:23:34.493Z - [WATCHER]: .nx/cache/d/daemon.log was modified
[NX Daemon Server] - 2024-02-23T07:23:34.493Z - Time taken for 'changed-projects' 0.023333000000093307ms
[NX Daemon Server] - 2024-02-23T07:23:34.546Z - [REQUEST]: Updated workspace context based on watched changes, recomputing project graph...
[NX Daemon Server] - 2024-02-23T07:23:34.546Z - [REQUEST]: .nx/cache/d/daemon.log,.nx/cache/project-graph.json,.nx/cache/file-map.json,.nx/cache/jest.hash
[NX Daemon Server] - 2024-02-23T07:23:34.546Z - [REQUEST]: .nx/cache/project-graph.json~d4536c1f,.nx/cache/file-map.json~d4536c1f
[NX Daemon Server] - 2024-02-23T07:23:34.706Z - [WATCHER]: .nx/cache/d/daemon.log was modified
[NX Daemon Server] - 2024-02-23T07:23:34.707Z - [WATCHER]: .nx/cache/d/daemon.log was modified
[NX Daemon Server] - 2024-02-23T07:23:34.707Z - Time taken for 'hash changed files from watcher' 1.0797919999999976ms
[NX Daemon Server] - 2024-02-23T07:23:34.750Z - [WATCHER]: 2 file(s) created or restored, 2 file(s) modified, 2 file(s) deleted
[NX Daemon Server] - 2024-02-23T07:23:34.750Z - Time taken for 'changed-projects' 0.03479199999992488ms
Package Manager Version
pnpm 8.15.1
Operating System
- [ ] macOS
- [X] Linux
- [ ] Windows
- [ ] Other (Please specify)
Additional Information
No response
Same issue here, v18.3.4, trying to serve my applications in containers I get:
Watch error: connect ENOENT /tmp/540f365a5f158298547b/d.sock
Watch error: Daemon closed the connection
Same setup works on v17.1.3
Same issue for me - there are no other clues as to what might be happening...
tried deleting node_modules and reinstalling
tried deleting .nx folder
tried adding --skipNxCache
I am not using docker
My node service won't start up.
I have a very large react application that runs just fine.
I am pretty sure that this was working on this same version previously, then I made some code changes a few days later and it stopped working.
EDIT:
I ran nx dameon to get logs - nothing seemed odd there
Then the project magically started up and worked the next time 🤷
yarn start:api
yarn run v1.22.21
$ nx serve api
> nx run api:serve:development
Watch error: connect ENOENT /var/folders/gj/hqrt8g957bgfpvsn87316_w80000gn/T/105c91fd379e4017ed70/d.sock
Watch error: Daemon closed the connection
$ nx report
NX Report complete - copy this into the issue template
Node : 20.10.0
OS : darwin-arm64
yarn : 1.22.21
nx : 18.3.4
@nx/js : 18.3.4
@nx/jest : 18.3.4
@nx/linter : 18.3.4
@nx/eslint : 18.3.4
@nx/workspace : 18.3.4
@nx/cypress : 18.3.4
@nx/devkit : 18.3.4
@nx/esbuild : 18.3.4
@nx/eslint-plugin : 18.3.4
@nx/express : 18.3.4
@nx/next : 18.3.4
@nx/node : 18.3.4
@nx/playwright : 18.3.4
@nx/plugin : 18.3.4
@nx/react : 18.3.4
@nx/storybook : 18.3.4
@nrwl/tao : 18.3.4
@nx/vite : 18.3.4
@nx/web : 18.3.4
@nx/webpack : 18.3.4
typescript : 5.4.5
---------------------------------------
Registered Plugins:
@nx/next/plugin
@nx/eslint/plugin
Hello! Thanks for taking the time to report this issue :)
@Nevation it looks like your docker container is simply running out of memory. The default for Node.js is 512 MB, is it unreasonable to think your app is simply hitting its limits in terms of resources in the container?
Have you tried increasing the memory via --max-old-space-size?
If you determine that this memory usage is not reasonable and there really is an nx bug, please provide a minimal reproduction that I can clone and have a look at. Sadly I don't have time to construct reproductions for every issue myself so I need your cooperation here 🙏 A memory profile from when the issue happens would also be useful.
@niallconaghan @paustint it looks like you're facing somewhat different errors. The directory where Nx thinks the socket is located doesn't exist anymore, thus resulting in the ENOENT.
It might be an issue with the temp folder where it's located for both of you, maybe it was deleted by docker or the OS in the meantime. If the issue persists, you can control where the socket is located by setting the NX_DAEMON_SOCKET_DIR environment variable. More information here: https://nx.dev/concepts/nx-daemon#customizing-the-socket-location
The only way I could get rid of that error „Daemon closed the connection“ was by running nx reset beforehand. So the scripts in package.json that run backend services in docker look like that: "loyalty:dev:docker": "nx reset && nx serve loyalty", "order:dev:docker": "nx reset && nx serve order", "payment:dev:docker": "nx reset && nx serve payment", "warehouse:dev:docker": "nx reset && nx serve warehouse"
if you wish to replicate this error you could try out this repo and remove „nx reset“. I guess they will launch at first, but if the containers are restartd then the error will appear. https://github.com/vturbin/saga-workshop/tree/main
This issue has been automatically marked as stale because more information has not been provided within 7 days. It will be closed in 21 days if no information is provided. If information has been provided, please reply to keep it active. Thanks for being a part of the Nx community! 🙏
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
I'm experiencing the same issue. It appears that Nx stores these runs in the cloud cache and incorrectly marks them as successful. As a result, it continuously retrieves the failed runs from the cache, causing us to get stuck.
A workaround that has been effective for us is using the --skip-nx-cache flag.
While this solution works, I believe we should investigate why clearly failed runs are being stored as successful in the cloud cache.
This issue has been automatically marked as stale because more information has not been provided within 7 days. It will be closed in 21 days if no information is provided. If information has been provided, please reply to keep it active. Thanks for being a part of the Nx community! 🙏
This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.