bunchee icon indicating copy to clipboard operation
bunchee copied to clipboard
verified publisher icon huozhi

out of memory if too many conditional exports

Open himself65 opened this issue 1 year ago • 11 comments 

/Users/himself65/Library/Caches/fnm_multishells/52995_1705525934746/bin/pnpm run build

> [email protected] build /Users/himself65/Code/LlamaIndexTS/packages/core
> bunchee


<--- Last few GCs --->

[60488:0x130008000]    18691 ms: Mark-Compact 4044.0 (4136.8) -> 4032.5 (4140.3) MB, 515.00 / 0.00 ms  (average mu = 0.531, current mu = 0.023) allocation failure; scavenge might not succeed
[60488:0x130008000]    19765 ms: Mark-Compact 4049.4 (4141.6) -> 4035.8 (4144.1) MB, 1034.92 / 0.00 ms  (average mu = 0.273, current mu = 0.036) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0x102b20bf4 node::Abort() [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
 2: 0x102b20ddc node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
 3: 0x102ca4da8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
 4: 0x102e796e8 v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
 5: 0x102e781c4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
 6: 0x102e6e9dc v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
 7: 0x102e6f23c v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
 8: 0x102e54240 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
 9: 0x10323be70 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
10: 0x103598c44 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
11: 0x103567ad4 Builtins_MapConstructor [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
12: 0x10350d8fc Builtins_JSBuiltinsConstructStub [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
13: 0x108debd84 
14: 0x108de33a4 
15: 0x108df20b8 
16: 0x108df1d70 
17: 0x108de21d0 
18: 0x108ddef64 
19: 0x108b2e5fc 
20: 0x108a15004 
21: 0x108de6a64 
22: 0x108df0848 
23: 0x108de5108 
24: 0x108ddf098 
25: 0x108b88130 
26: 0x108ddf3c8 
27: 0x1089deb3c 
28: 0x1089e0760 
29: 0x108a15004 
30: 0x108de6a64 
31: 0x108de5420 
32: 0x108ddf098 
33: 0x108ae13dc 
34: 0x1089e1370 
35: 0x108a15004 
36: 0x108de6a64 
37: 0x108de5420 
38: 0x108ddf098 
39: 0x108de96f4 
40: 0x108b88260 
41: 0x108ddf3c8 
42: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
43: 0x108de5274 
44: 0x108deb1f0 
45: 0x108ddf0e4 
46: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
47: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
48: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
49: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
50: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
51: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
52: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
53: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
54: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
55: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
56: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
57: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
58: 0x1035103e4 Builtins_InterpreterEntryTrampoline [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
59: 0x108950f14 
60: 0x1035f4fb8 Builtins_PromiseFulfillReactionJob [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
61: 0x103536b94 Builtins_RunMicrotasks [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
62: 0x10350e3f4 Builtins_JSRunMicrotasksEntry [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
63: 0x102de69f0 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
64: 0x102de6edc v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
65: 0x102de70b8 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
66: 0x102e0e284 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
67: 0x102e0ea20 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
68: 0x102a50c64 node::InternalCallbackScope::Close() [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
69: 0x102a507c4 node::InternalCallbackScope::~InternalCallbackScope() [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
70: 0x102b24bdc node::fs::FileHandle::CloseReq::Resolve() [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
71: 0x102b3d9c8 node::fs::FileHandle::ClosePromise()::$_0::__invoke(uv_fs_s*) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
72: 0x102b1bcf0 node::MakeLibuvRequestCallback<uv_fs_s, void (*)(uv_fs_s*)>::Wrapper(uv_fs_s*) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
73: 0x1034ecb64 uv__work_done [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
74: 0x1034f05b4 uv__async_io [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
75: 0x10350268c uv__io_poll [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
76: 0x1034f0b78 uv_run [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
77: 0x102a51754 node::SpinEventLoopInternal(node::Environment*) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
78: 0x102b608d8 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
79: 0x102b60674 node::NodeMainInstance::Run() [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
80: 0x102aeb030 node::Start(int, char**) [/Users/himself65/Library/Application Support/fnm/node-versions/v20.10.0/installation/bin/node]
81: 0x1891d10e0 start [/usr/lib/dyld]
 ELIFECYCLE  Command failed.

Process finished with exit code 134 (interrupted by signal 6:SIGABRT)

{
  "name": "llamaindex",
  "version": "0.0.46",
  "license": "MIT",
  "dependencies": {
    "@anthropic-ai/sdk": "^0.9.1",
    "@aws-crypto/sha256-browser": "^5.2.0",
    "@datastax/astra-db-ts": "^0.1.2",
    "@mistralai/mistralai": "^0.0.7",
    "@notionhq/client": "^2.2.14",
    "@pinecone-database/pinecone": "^1.1.2",
    "@xenova/transformers": "^2.10.0",
    "assemblyai": "^4.0.0",
    "chromadb": "^1.7.3",
    "file-type": "^18.7.0",
    "js-tiktoken": "^1.0.8",
    "lodash": "^4.17.21",
    "mammoth": "^1.6.0",
    "md-utils-ts": "^2.0.0",
    "mongodb": "^6.3.0",
    "notion-md-crawler": "^0.0.2",
    "openai": "^4.20.1",
    "papaparse": "^5.4.1",
    "pdfjs-dist": "4.0.269",
    "pg": "^8.11.3",
    "pgvector": "^0.1.5",
    "portkey-ai": "^0.1.16",
    "rake-modified": "^1.0.8",
    "replicate": "^0.21.1",
    "string-strip-html": "^13.4.3",
    "wink-nlp": "^1.14.3"
  },
  "devDependencies": {
    "@types/jest": "^29.5.11",
    "@types/lodash": "^4.14.202",
    "@types/node": "^18.19.6",
    "@types/papaparse": "^5.3.14",
    "@types/pg": "^8.10.9",
    "bunchee": "^4.4.0",
    "node-stdlib-browser": "^1.2.0",
    "typescript": "^5.3.3"
  },
  "engines": {
    "node": ">=18.0.0"
  },
  "types": "./dist/index.d.ts",
  "main": "./dist/index.js",
  "exports": {
    ".": {
      "types": "./dist/index.d.mts",
      "edge-light": "./dist/index.edge-light.mjs",
      "import": "./dist/index.mjs",
      "require": "./dist/index.js"
    },
    "./storage/FileSystem": {
      "types": "./dist/storage/FileSystem.d.mts",
      "edge-light": "./dist/storage/FileSystem.edge-light.mjs",
      "import": "./dist/storage/FileSystem.mjs",
      "require": "./dist/storage/FileSystem.js"
    },
    "./ChatEngine": {
      "types": "./dist/ChatEngine.d.mts",
      "edge-light": "./dist/ChatEngine.edge-light.mjs",
      "import": "./dist/ChatEngine.mjs",
      "require": "./dist/ChatEngine.js"
    },
    "./ChatHistory": {
      "types": "./dist/ChatHistory.d.mts",
      "edge-light": "./dist/ChatHistory.edge-light.mjs",
      "import": "./dist/ChatHistory.mjs",
      "require": "./dist/ChatHistory.js"
    },
    "./constants": {
        "types": "./dist/constants.d.mts",
        "edge-light": "./dist/constants.edge-light.mjs",
        "import": "./dist/constants.mjs",
        "require": "./dist/constants.js"
        },
    "./GlobalsHelper": {
        "types": "./dist/GlobalsHelper.d.mts",
        "edge-light": "./dist/GlobalsHelper.edge-light.mjs",
        "import": "./dist/GlobalsHelper.mjs",
        "require": "./dist/GlobalsHelper.js"

    },
    "./Node": {
        "types": "./dist/Node.d.mts",
        "edge-light": "./dist/Node.edge-light.mjs",
        "import": "./dist/Node.mjs",
        "require": "./dist/Node.js"

    },
    "./OutputParser": {
        "types": "./dist/OutputParser.d.mts",
        "edge-light": "./dist/OutputParser.edge-light.mjs",
        "import": "./dist/OutputParser.mjs",
        "require": "./dist/OutputParser.js"
    },
    "./Prompt": {
        "types": "./dist/Prompt.d.mts",
        "edge-light": "./dist/Prompt.edge-light.mjs",
        "import": "./dist/Prompt.mjs",
        "require": "./dist/Prompt.js"
    },
    "./PromptHelper": {
        "types": "./dist/PromptHelper.d.mts",
        "edge-light": "./dist/PromptHelper.edge-light.mjs",
        "import": "./dist/PromptHelper.mjs",
        "require": "./dist/PromptHelper.js"
    },
    "./QueryEngine": {
        "types": "./dist/QueryEngine.d.mts",
        "edge-light": "./dist/QueryEngine.edge-light.mjs",
        "import": "./dist/QueryEngine.mjs",
        "require": "./dist/QueryEngine.js"
    },
    "./QuestionGenerator": {
        "types": "./dist/QuestionGenerator.d.mts",
        "edge-light": "./dist/QuestionGenerator.edge-light.mjs",
        "import": "./dist/QuestionGenerator.mjs",
        "require": "./dist/QuestionGenerator.js"
    },
    "./Response": {
        "types": "./dist/Response.d.mts",
        "edge-light": "./dist/Response.edge-light.mjs",
        "import": "./dist/Response.mjs",
        "require": "./dist/Response.js"
    },
    "./Retriever": {
        "types": "./dist/Retriever.d.mts",
        "edge-light": "./dist/Retriever.edge-light.mjs",
        "import": "./dist/Retriever.mjs",
        "require": "./dist/Retriever.js"
    },
    "./ServiceContext": {
        "types": "./dist/ServiceContext.d.mts",
        "edge-light": "./dist/ServiceContext.edge-light.mjs",
        "import": "./dist/ServiceContext.mjs",
        "require": "./dist/ServiceContext.js"
    },
    "./TextSplitter": {
        "types": "./dist/TextSplitter.d.mts",
        "edge-light": "./dist/TextSplitter.edge-light.mjs",
        "import": "./dist/TextSplitter.mjs",
        "require": "./dist/TextSplitter.js"
    },
    "./Tool": {
        "types": "./dist/Tool.d.mts",
        "edge-light": "./dist/Tool.edge-light.mjs",
        "import": "./dist/Tool.mjs",
        "require": "./dist/Tool.js"
    },
    "./readers/AssemblyAI": {
      "types": "./dist/readers/AssemblyAI.d.mts",
      "edge-light": "./dist/readers/AssemblyAI.edge-light.mjs",
      "import": "./dist/readers/AssemblyAI.mjs",
      "require": "./dist/readers/AssemblyAI.js"
    },
    "./readers/base": {
      "types": "./dist/readers/base.d.mts",
      "edge-light": "./dist/readers/base.edge-light.mjs",
      "import": "./dist/readers/base.mjs",
      "require": "./dist/readers/base.js"
    },
    "./readers/CSVReader": {
      "types": "./dist/readers/CSVReader.d.mts",
      "edge-light": "./dist/readers/CSVReader.edge-light.mjs",
      "import": "./dist/readers/CSVReader.mjs",
      "require": "./dist/readers/CSVReader.js"
    },
    "./readers/DocxReader": {
      "types": "./dist/readers/DocxReader.d.mts",
      "edge-light": "./dist/readers/DocxReader.edge-light.mjs",
      "import": "./dist/readers/DocxReader.mjs",
      "require": "./dist/readers/DocxReader.js"
    },
    "./readers/HTMLReader": {
      "types": "./dist/readers/HTMLReader.d.mts",
      "edge-light": "./dist/readers/HTMLReader.edge-light.mjs",
      "import": "./dist/readers/HTMLReader.mjs",
      "require": "./dist/readers/HTMLReader.js"
    },
    "./readers/ImageReader": {
      "types": "./dist/readers/ImageReader.d.mts",
      "edge-light": "./dist/readers/ImageReader.edge-light.mjs",
      "import": "./dist/readers/ImageReader.mjs",
      "require": "./dist/readers/ImageReader.js"
    },
    "./readers/MarkdownReader": {
      "types": "./dist/readers/MarkdownReader.d.mts",
      "edge-light": "./dist/readers/MarkdownReader.edge-light.mjs",
      "import": "./dist/readers/MarkdownReader.mjs",
      "require": "./dist/readers/MarkdownReader.js"
    },
    "./readers/NotionReader": {
      "types": "./dist/readers/NotionReader.d.mts",
      "edge-light": "./dist/readers/NotionReader.edge-light.mjs",
      "import": "./dist/readers/NotionReader.mjs",
      "require": "./dist/readers/NotionReader.js"
    },
    "./readers/PDFReader": {
      "types": "./dist/readers/PDFReader.d.mts",
      "edge-light": "./dist/readers/PDFReader.edge-light.mjs",
      "import": "./dist/readers/PDFReader.mjs",
      "require": "./dist/readers/PDFReader.js"
    },
    "./readers/SimpleDirectoryReader": {
      "types": "./dist/readers/SimpleDirectoryReader.d.mts",
      "edge-light": "./dist/readers/SimpleDirectoryReader.edge-light.mjs",
      "import": "./dist/readers/SimpleDirectoryReader.mjs",
      "require": "./dist/readers/SimpleDirectoryReader.js"
    },
    "./readers/SimpleMongoReader": {
      "types": "./dist/readers/SimpleMongoReader.d.mts",
      "edge-light": "./dist/readers/SimpleMongoReader.edge-light.mjs",
      "import": "./dist/readers/SimpleMongoReader.mjs",
      "require": "./dist/readers/SimpleMongoReader.js"
    },
    "./environments": {
      "types": "./dist/environments.d.mts",
      "edge-light": "./dist/environments.edge-light.mjs",
      "import": "./dist/environments.mjs",
      "require": "./dist/environments.js"
    },
    "./examples/*": "./examples/*"
  },
  "files": [
    "dist",
    "examples",
    "src",
    "types",
    "CHANGELOG.md"
  ],
  "repository": {
    "type": "git",
    "url": "https://github.com/run-llama/LlamaIndexTS.git",
    "directory": "packages/core"
  },
  "scripts": {
    "lint": "eslint .",
    "test": "jest",
    "build": "bunchee",
    "dev": "bunchee -w"
  }
}

himself65 avatar Jan 17 '24 23:01 himself65

Do you have a reproduction link? did'nt see bunchee is setup in that repo

huozhi avatar Jan 17 '24 23:01 huozhi

Do you have a reproduction link? did'nt see bunchee is setup in that repo

im still working on this

himself65 avatar Jan 17 '24 23:01 himself65

but you can overwrite the package.json and see it

himself65 avatar Jan 17 '24 23:01 himself65

Possible you bundled sth re not in the dependencies or peer dependencies, it happened before when some users were trying to bundle webpack but actually webpack should be external

huozhi avatar Jan 21 '24 01:01 huozhi

see https://github.com/run-llama/LlamaIndexTS/pull/434

himself65 avatar Jan 24 '24 21:01 himself65

Thanks I can repro it now, will investigate which part is having mem leak 👁️‍🗨️

One possible is probably due to rollup production optimization that causes a lot of memory, or it's using too much mem in ts generation plugin

huozhi avatar Jan 25 '24 22:01 huozhi

Looks like if there's no types generation they memory allocation will be fine without specifying extra memory, sth wrong with the ts generation.

huozhi avatar Jan 28 '24 00:01 huozhi

did some debugging using the reproducible example and for each new export the time of the other export bundle increases significantly.

I tried to run it synchronously then decreased it from 30s to 1.50s per exported module (max 4.50s big ones), still playing with this

image image

EmanuelCampos avatar Feb 12 '24 15:02 EmanuelCampos

This is still happening on our case especially when increasing the exports, sadly I'm considering to switch to other bundler

himself65 avatar Feb 22 '24 06:02 himself65

I'm rewriting the exports parser and in the shorter future we gonna generate less type files for your case which could help a bit on the memory. I can't commit a time that can fix this soon since the resource from me on this is pretty limited atm.

huozhi avatar Feb 22 '24 11:02 huozhi

So the generating more types file is not the main problem. Seems ts plugin will consume too much memory anyway. I might need to move those work to a separate worker

huozhi avatar Mar 03 '24 23:03 huozhi

Tested with the repro, it's resolved with 5.0.0-beta.2

huozhi avatar Mar 09 '24 20:03 huozhi

@himself65 [email protected] improved more about the types generation perf for multi exports! 🙏

huozhi avatar Mar 20 '24 23:03 huozhi