[bug] knowledge base给我玩GC了

[meokey]

刚才暴力测试了一下，给一个kb塞了好多文件，应该大于等于100MB了，终于GC了，呵呵 😢 💀

··· <--- Last few GCs --->

[1:0x73a2aa0] 16845520 ms: Mark-Compact 4041.8 (4133.0) -> 4027.7 (4135.0) MB, 6641.42 / 0.00 ms (average mu = 0.134, current mu = 0.068) allocation failure; scavenge might not succeed [1:0x73a2aa0] 16850957 ms: Mark-Compact 4043.6 (4135.0) -> 4029.4 (4137.0) MB, 4935.94 / 0.00 ms (average mu = 0.115, current mu = 0.092) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc98550 node::Abort() [node] 2: 0xb700a7 [node] 3: 0xebae80 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node] 4: 0xebb167 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node] 5: 0x10cc7d5 [node] 6: 0x10ccd64 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node] 7: 0x10e3c54 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node] 8: 0x10e446c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node] 9: 0x113d22c v8::internal::MinorGCJob::Task::RunInternal() [node] 10: 0xd123c6 [node] 11: 0xd14f7f node::PerIsolatePlatformData::FlushForegroundTasksInternal() [node] 12: 0x187af93 [node] 13: 0x188f95b [node] 14: 0x187bcb7 uv_run [node] 15: 0xbb3ba6 node::SpinEventLoopInternal(node::Environment*) [node] 16: 0xce8036 node::NodeMainInstance::Run() [node] 17: 0xc5112f node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResultImpl const*) [node] 18: 0xc54ab2 node::Start(int, char**) [node] 19: 0x7f85139ad1ca [/lib/x86_64-linux-gnu/libc.so.6] 20: 0x7f85139ad285 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6] 21: 0xbb004e _start [node] ···

[sugarforever]

刚才暴力测试了一下，给一个kb塞了好多文件，应该大于等于100MB了，终于GC了，呵呵 😢 💀

··· <--- Last few GCs --->

[1:0x73a2aa0] 16845520 ms: Mark-Compact 4041.8 (4133.0) -> 4027.7 (4135.0) MB, 6641.42 / 0.00 ms (average mu = 0.134, current mu = 0.068) allocation failure; scavenge might not succeed [1:0x73a2aa0] 16850957 ms: Mark-Compact 4043.6 (4135.0) -> 4029.4 (4137.0) MB, 4935.94 / 0.00 ms (average mu = 0.115, current mu = 0.092) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc98550 node::Abort() [node] 2: 0xb700a7 [node] 3: 0xebae80 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node] 4: 0xebb167 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node] 5: 0x10cc7d5 [node] 6: 0x10ccd64 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node] 7: 0x10e3c54 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node] 8: 0x10e446c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node] 9: 0x113d22c v8::internal::MinorGCJob::Task::RunInternal() [node] 10: 0xd123c6 [node] 11: 0xd14f7f node::PerIsolatePlatformData::FlushForegroundTasksInternal() [node] 12: 0x187af93 [node] 13: 0x188f95b [node] 14: 0x187bcb7 uv_run [node] 15: 0xbb3ba6 node::SpinEventLoopInternal(node::Environment*) [node] 16: 0xce8036 node::NodeMainInstance::Run() [node] 17: 0xc5112f node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResultImpl const*) [node] 18: 0xc54ab2 node::Start(int, char**) [node] 19: 0x7f85139ad1ca [/lib/x86_64-linux-gnu/libc.so.6] 20: 0x7f85139ad285 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6] 21: 0xbb004e _start [node] ···

😅😂🤣

目前KB创建还是有性能隐患的，因为我知道目前文档分块都是加载到内存里，一次性写入向量存储的。这里很可能会溢出。