remix-run
[Bug]: JavaScript heap out of memory
Which Remix packages are impacted?
remix(Remix core)@remix-run/dev@remix-run/node@remix-run/react@remix-run/serve@remix-run/vercel
What version of Remix are you using?
v1.1.1
What version of Node are you using? Minimum supported version is 14.
v16.13.1
Steps to Reproduce
It's just a regular remix app and sometimes when saving something in the codebase the server crashes and gives me the error pasted below, relating to JS memory issues. Looks like something to do with esbuild but I'm not really sure, just wondering if anyone else ever has this?
Expected Behavior
Not crash
Actual Behavior
<--- Last few GCs --->
[37477:0x130008000] 7075356 ms: Mark-sweep 4025.7 (4134.4) -> 4015.9 (4140.4) MB, 3782.7 / 0.0 ms (average mu = 0.694, current mu = 0.007) allocation failure scavenge might not succeed
[37477:0x130008000] 7095099 ms: Mark-sweep 4034.0 (4142.4) -> 4023.3 (4147.9) MB, 4517.8 / 0.0 ms (average mu = 0.742, current mu = 0.771) allocation failure scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
1: 0x102d1846c node::Abort() [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
2: 0x102d185f4 node::errors::TryCatchScope::~TryCatchScope() [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
3: 0x102e667b4 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
4: 0x102e66748 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
5: 0x102fe9c08 v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
6: 0x102fe8728 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
7: 0x102ff3a04 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
8: 0x102ff3a98 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
9: 0x102fbece4 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
10: 0x1031f5638 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::AllocationType) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
11: 0x1031f5fe0 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
12: 0x1031f5f54 v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
13: 0x1031ba59c v8::internal::JSObject::SetNormalizedProperty(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
14: 0x1031d7268 v8::internal::LookupIterator::TransitionToAccessorPair(v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
15: 0x1031d6b50 v8::internal::LookupIterator::TransitionToAccessorProperty(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
16: 0x1031b53d4 v8::internal::JSObject::DefineAccessor(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
17: 0x1031b50dc v8::internal::JSReceiver::ValidateAndApplyPropertyDescriptor(v8::internal::Isolate*, v8::internal::LookupIterator*, bool, v8::internal::PropertyDescriptor*, v8::internal::PropertyDescriptor*, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::Handle<v8::internal::Name>) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
18: 0x1031b4428 v8::internal::JSReceiver::OrdinaryDefineOwnProperty(v8::internal::LookupIterator*, v8::internal::PropertyDescriptor*, v8::Maybe<v8::internal::ShouldThrow>) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
19: 0x1031b3f64 v8::internal::JSReceiver::OrdinaryDefineOwnProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSObject>, v8::internal::PropertyKey const&, v8::internal::PropertyDescriptor*, v8::Maybe<v8::internal::ShouldThrow>) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
20: 0x1031b3b74 v8::internal::JSReceiver::DefineOwnProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDescriptor*, v8::Maybe<v8::internal::ShouldThrow>) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
21: 0x1031b3a0c v8::internal::JSReceiver::DefineProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
22: 0x102ef1164 v8::internal::Builtin_ObjectDefineProperty(int, unsigned long*, v8::internal::Isolate*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
23: 0x10361042c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
24: 0x1082b4198
25: 0x1082b37e4
26: 0x1035a3f78 Builtins_InterpreterEntryTrampoline [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
27: 0x107e38a4c
28: 0x107731ddc
29: 0x107922624
30: 0x107f51e98
31: 0x107a714cc
32: 0x107cab644
33: 0x1079f1d20
34: 0x1079f1be8
35: 0x1079f1be8
36: 0x1079f1be8
37: 0x1079f1be8
38: 0x1079f1be8
39: 0x1079f1be8
40: 0x10784b71c
41: 0x107cab644
42: 0x1079dcba0
43: 0x107acb0e0
44: 0x107acb40c
45: 0x107e20784
46: 0x1079d61a0
47: 0x107a72e0c
48: 0x107e20a08
49: 0x1079d61a0
50: 0x107ad59b4
51: 0x107b70e74
52: 0x1079dcf34
53: 0x107bae498
54: 0x107b15d1c
55: 0x107b70940
56: 0x107af1ef4
57: 0x107832bec
58: 0x1035a1d6c Builtins_JSEntryTrampoline [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
59: 0x1035a1a04 Builtins_JSEntry [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
60: 0x102f76488 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
61: 0x102f75b1c v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
62: 0x102e830c0 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
63: 0x102c68280 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
64: 0x102c7d4ac node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
65: 0x102d1c628 node::fs::FSReqCallback::Reject(v8::Local<v8::Value>) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
66: 0x102d1cdf4 node::fs::FSReqAfterScope::Reject(uv_fs_s*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
67: 0x102d1cf1c node::fs::AfterStat(uv_fs_s*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
68: 0x1035823b0 uv__work_done [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
69: 0x103585b10 uv__async_io [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
70: 0x103597824 uv__io_poll [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
71: 0x103585fa0 uv_run [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
72: 0x102c68e00 node::SpinEventLoop(node::Environment*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
73: 0x102d51638 node::NodeMainInstance::Run(int*, node::Environment*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
74: 0x102d51304 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
75: 0x102cea2ec node::Start(int, char**) [/Users/jclackett/.fnm/node-versions/v16.13.1/installation/bin/node]
76: 0x1075f50f4
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x140000b4c28)
runtime/sema.go:56 +0x38
sync.(*WaitGroup).Wait(0x140000b4c20)
sync/waitgroup.go:130 +0xa4
main.runService(0x1)
github.com/evanw/esbuild/cmd/esbuild/service.go:138 +0x494
main.main()
github.com/evanw/esbuild/cmd/esbuild/main.go:203 +0x1f0
goroutine 18 [chan receive]:
main.runService.func1(0x140000cc140, 0x140000b4c20)
github.com/evanw/esbuild/cmd/esbuild/service.go:66 +0x3c
created by main.runService
github.com/evanw/esbuild/cmd/esbuild/service.go:64 +0x1a8
goroutine 19 [chan receive]:
main.(*serviceType).sendRequest(0x140000cc140, {0x102c3b320, 0x140044368a0})
github.com/evanw/esbuild/cmd/esbuild/service.go:163 +0x104
main.runService.func2(0x140000cc140)
github.com/evanw/esbuild/cmd/esbuild/service.go:92 +0x40
created by main.runService
github.com/evanw/esbuild/cmd/esbuild/service.go:89 +0x2b8
error Command failed with signal "SIGABRT".
I've also been receiving this issue randomly after working on in my project with the dev server running for a few hours.
@JClackett I've just tested it on your remix boilerplate and it used half the memory was using before (I had to remove the sentry imports though).
I used autocannon for that: autocannon -d 30 -c 50 http://localhost:3000/posts
Sentry might be doing some global initialization that hangs around and is re-called every time the require cache is purged (i.e every request in dev).
@viczam what version of Node you on? and what machine?
I'm using node 16 on an m1 Mac its not working using all latest remix deps.
I have found I get this issue also occasionally. But only on an old MacBook Pro with low specs. When I work on my later machine with better specs it doesn't happen 🤷🏼♂️
I am still getting this issue. Is there a way to change the scripts to allow more memory?
I found that the source of this error for me was importing a large (20mb) json file as
import Messages from '../messages.json' - the way I fixed it was making a new messages.server.ts file and doing a node file read instead.
import path from "path";
import fs from "fs/promises";
const messagePath = path.join(__dirname, "..", "messages.json");
export async function getMessages() {
const file = await fs.readFile(messagePath, "utf8");
return JSON.parse(file);
}
I guess this is happening because using an import is continually adding the json to memory and the garbage collector either isn't keeping up or isn't freeing up that memory at all. But the node file system read does. Just speculation as I stopped looking in to it once I found my fix.
Part of the issue is that Remix is constantly purging the require cache on every request (in "development"). This results in a lot of garbage and if you make enough requests quickly enough, the process will run out of memory before the GC can do its thing.
I have my own <LiveReload/> component, that calls an endpoint http://localhost:3000/build/__purge__ before it does the refresh. Then in my Express adapter, I simply add a handler for that route to do the purge instead of every request.
// LiveReload.tsx
if (event.type === "RELOAD") {
console.log("💿 Reloading window ...");
- window.location.reload();
+ fetch("/build/__purge__").finally(() => {
+ window.location.reload();
+ });
}
// server.js (Express adapter)
app.use('/build/__purge__', (req, res) => {
console.log('💿 Purging require cache')
purgeRequireCache()
return res.send('ok')
})
Here is the updated LiveReload component with @kiliman's changes.
process.env.NODE_ENV !== 'development'
? () => null
: function LiveReload({
port = Number(process.env.REMIX_DEV_SERVER_WS_PORT || 8002),
nonce = undefined,
}: {
port?: number;
nonce?: string;
}) {
const js = String.raw;
return (
<script
nonce={nonce}
suppressHydrationWarning
dangerouslySetInnerHTML={{
__html: js`
(() => {
let protocol = location.protocol === "https:" ? "wss:" : "ws:";
let host = location.hostname;
let socketPath = protocol + "//" + host + ":" + ${String(port)} + "/socket";
let ws = new WebSocket(socketPath);
ws.onmessage = (message) => {
let event = JSON.parse(message.data);
if (event.type === "LOG") {
console.log(event.message);
}
if (event.type === "RELOAD") {
console.log("💿 Reloading window ...");
window.location.reload();
fetch("/build/__purge__").finally(() => {
window.location.reload();
});
}
};
ws.onerror = (error) => {
console.log("Remix dev asset server web socket error:");
console.error(error);
};
})();
`,
}}
/>
);
};
In my express server I did:
if (MODE !== 'production') {
app.use((_req, _res, next) => {
// purgeRequireCache();
build = require('./build');
next();
});
app.use('/build/__purge__', (req, res) => {
console.log('💿 Purging require cache');
purgeRequireCache();
return res.send('ok');
});
}```
Getting this constantly on a large app, seems like after a couple save/LiveReload cycles, it blows up. In active development, it's happening every 5-10 minutes.
Setting NODE_OPTIONS="--max-old-space-size=8192" seems to have temporarily fixed this -- working for a couple hours now with no out of memory errors.
@vinceprofeta you’ll want to delete that first reload() call. It should only reload in the finally block.
This is literally the main major point of remix developer experience. Otherwise, remix is so awesome!
Yes I plan on submitting a PR to fix this. I’m flying back today from Remix Conf which was awesome.
@kiliman Using your LiveReload / express server updates definitely helps. One small issue I found is you can get a stale page in development if this happens:
- Load a page (so it's in the require cache)
- Close the page
- Make changes to the page and save, Remix rebuilds.
- Re-open the page, you still have the old version since the LiveReload component wasn't active during the update, so the
/build/__purge__handler never ran. At this point you either need to restart the server, or make a change to the cached page to trigger the purge.
In practice this won't happen often, since you usually have the page open during development, but it's an edge case to consider if you're submitting a PR.
Ah, interesting scenario. Perhaps what should happen is that remix watch should send the request to purge to the web server instead of the browser.
- Remix watch detects change and rebuilds
- Sends purge request to web server
- Sends web socket notification to browser to refresh
So even if browser is not active to refresh, the web server was already notified to purge and will serve new content when browser reconnects.
So if you use remix dev instead of a separate express server, purging it during the rebuild is pretty easy. Just move the purge call from https://github.com/remix-run/remix/blob/main/packages/remix-dev/cli/commands.ts#L271 to right before https://github.com/remix-run/remix/blob/main/packages/remix-dev/cli/commands.ts#L211, so that the cache is purged before the RELOAD signal is sent to the browser.
However, if you use a separate express server, I'm not sure the best way to clear that separate process's require cache from the remix watch command...
@kiliman do you know if there is some things going on to fix that bug ?
@dmarkow it seems the files are outdated now, could you update the fix you propose to the last updated files ? hard for me to understand !
that would be really nice...
thanks !
actually I tried something and it looks better, is my patch correct ?
@remix-run+dev+1.7.1.patch
diff --git a/node_modules/@remix-run/dev/dist/cli/commands.js b/node_modules/@remix-run/dev/dist/cli/commands.js
index 54f2d99..80eb2fa 100644
--- a/node_modules/@remix-run/dev/dist/cli/commands.js
+++ b/node_modules/@remix-run/dev/dist/cli/commands.js
@@ -228,6 +228,7 @@ async function watch(remixRootOrConfig, modeArg, callbacks) {
},
onRebuildFinish() {
+ purgeAppRequireCache(config$1.serverBuildPath);
log(`Rebuilt in ${prettyMs__default["default"](Date.now() - start)}`);
broadcast({
type: "RELOAD"
@@ -289,7 +290,6 @@ async function dev(remixRoot, modeArg, portArg) {
let app = express();
app.disable("x-powered-by");
app.use((_, __, next) => {
- purgeAppRequireCache(config$1.serverBuildPath);
next();
});
app.use(createApp(config$1.serverBuildPath, mode, config$1.publicPath, config$1.assetsBuildDirectory));
well I don't have any such problem since my patch, so it might be correct ! I'll open a PR to move things on, and see with the comments if my patch IS correct 😄
Happening on 1.7.5 too, remix is really bad at this. Development experience with this issue is terrible.
On every page load memory usage grows by a few megabytes, consistently. I've been trying to figure it out using Node's heap dumps, but it's just to much in there that I cannot figure it out.
However, I found a lot of identical cases of… I think it's cached code.
Trying to dig down in the cache I can see a lot of cached node_module files in there.
Is there some cache built-in somewhere that isn't flushed or reused on a reload so it just sticks around forever? I don't know, and I don't understand what I'm looking at.
If you want to have a look, run node --inspect ./node_modules/.bin/remix dev and open a Chromium-based browser to chrome://inspect and make memory snapshots. Perhaps someone else can make sense of this.
