11ty
Out of memory error with very large pagination
Describe the bug While testing updates irt to issue https://github.com/11ty/eleventy/issues/2360, I run out of memory somewhere between 30 and 50K pages. Here's my output:
ray@Hoth:~/projects/eleventy-demos/twitter_test$ NODE_OPTIONS=--max_old_space_size=8192 npx @11ty/[email protected] --quiet
size of tweets 50000
<--- Last few GCs --->
[8880:0x4977770] 35535 ms: Mark-sweep (reduce) 8189.6 (8221.0) -> 8189.2 (8222.0) MB, 849.9 / 0.0 ms (average mu = 0.089, current mu = 0.006) allocation failure scavenge might not succeed
[8880:0x4977770] 36478 ms: Mark-sweep (reduce) 8190.1 (8224.8) -> 8190.0 (8224.8) MB, 939.4 / 0.0 ms (average mu = 0.046, current mu = 0.004) allocation failure scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xa389b0 node::Abort() [node]
2: 0x96e0af node::FatalError(char const*, char const*) [node]
3: 0xbb7a4e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xbb7dc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xd73fd5 [node]
6: 0xd74b5f [node]
7: 0xd8299b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
8: 0xd8655c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
9: 0xd4bd2d v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
10: 0xd47f89 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [node]
11: 0xd48044 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [node]
12: 0xed58ce [node]
13: 0xedb164 [node]
14: 0xedea18 [node]
15: 0xf6e3ed v8::internal::JSArray::SetLength(v8::internal::Handle<v8::internal::JSArray>, unsigned int) [node]
16: 0xee30d5 v8::internal::ArrayConstructInitializeElements(v8::internal::Handle<v8::internal::JSArray>, v8::internal::Arguments<(v8::internal::ArgumentsType)1>*) [node]
17: 0x1074825 v8::internal::Runtime_NewArray(int, unsigned long*, v8::internal::Isolate*) [node]
18: 0x14
To Reproduce
Everything but my raw data is here: https://github.com/cfjedimaster/eleventy-demos/tree/master/twitter_test
I can share my twitter.js file which was too big to commit to GH.
Note - I was able to get 40k files working.
Yeah can you share that file? Not sure what the best way to send it is though 😅
I’d love to run this test locally
In the same repo, grab tweet_data/tweet.zip, then unzip it. Note that I'm testing by modifying the slice in _data/tweets.js.
We get this, too. Can supply additional info if the team is having a tough time tracking down a solution. I'm assuming this is tied to the master issue over at https://github.com/11ty/eleventy/issues/2226?
