Node process exit when adding some references fields filters

[Striptik]

Expected behavior

The expected behavior is to get the references filter working for all collections in all collections.

Actual behavior

Some collections have references fields on others collections (often large amount of data) and when triyng to add a filter with this references fields, after a few times, i get :

• 'Data cannot be reach' on the browser,
• On the browser console a Failed to load resource: the server responded with a status of 504 (GATEWAY_TIMEOUT)
• The node process exit with a FATAL ERROR due to JavaScript heap out of memory
• The API reboot making all the stack unavailable during 2 minutes

I tried to increase the size of the memory allocated to the node process with the option --max-old-space-size without success.

Failure Logs

The Node JS Failure log is

<--- Last few GCs --->

[25176:0x103000000] 175050 ms: Mark-sweep 1492.9 (1566.7) -> 1492.7 (1535.7) MB, 1287.0 / 0.0 ms last resort GC in old space requested [25176:0x103000000] 176334 ms: Mark-sweep 1492.7 (1535.7) -> 1492.7 (1535.7) MB, 1284.4 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3ac7b79a5749 <JSObject> 1: push(this=0x3ac7a157ac59 <JSArray[60167]>) 2: /* anonymous /(aka / anonymous */) [/..../node_modules/mongodb/lib/operations/cursor_ops.js:226] [bytecode=0x3ac7eb16d639 offset=292](this=0x3ac718e822d1 ,err=0x3ac718e82201 ,doc=0x3ac795dd0ee9 <Object map = 0x3ac7d1ac0cb9>) 3: handleCallback(aka handleCallback) [/.../api/node...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: node::Abort() [/usr/local/bin/node] 2: node::FatalException(v8::Isolate*, v8::Localv8::Value, v8::Localv8::Message) [/usr/local/bin/node] 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node] 4: v8::internal::Factory::NewUninitializedFixedArray(int) [/usr/local/bin/node] 5: v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedObjectElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)2> >::ConvertElementsWithCapacity(v8::internal::Handlev8::internal::JSObject, v8::internal::Handlev8::internal::FixedArrayBase, v8::internal::ElementsKind, unsigned int, unsigned int, unsigned int, int) [/usr/local/bin/node] 6: v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedObjectElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)2> >::GrowCapacityAndConvertImpl(v8::internal::Handlev8::internal::JSObject, unsigned int) [/usr/local/bin/node] 7: v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedObjectElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)2> >::Add(v8::internal::Handlev8::internal::JSObject, unsigned int, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, unsigned int) [/usr/local/bin/node] 8: v8::internal::JSObject::AddDataElement(v8::internal::Handlev8::internal::JSObject, unsigned int, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow) [/usr/local/bin/node] 9: v8::internal::Runtime_SetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node] 10: 0x32cb009842fd

Context

• Package Version: 2.14.0
• Express Version: 4.16.2
• Mongoose Version: 5.2.14
• MongoDB Version: 3.1.6 (package.json) | 4.0.2 (remote DB)

[arnaudbesnier]

Hi @Striptik, thanks for the "heap out of memory" error/stack trace.

Can you explain a bit more on which schema this error happens and what are the relations between them? Also it is not clear to me which request to the Liana API generate such errors, did you isolate them? In short the more detailed your configuration / error experience, the faster we'll dig into this.

Thanks for your help!

[Striptik]

Hi @arnaudbesnier,

This error happens when I filter my data, right from a collection. I add a reference field filter in some collection, the reference field type (string, number, datetime...) does not change anything.

Most of the time, these errors appeared on big collections, and the refered collections is big too. e.g.:
Customer (Collection) => lastAddressUsed (Address collection referred field on customer) => postalCode (field in address Collection) IS 75001

OR

Order (Collection) => customerId (Customer collection referred field on order) => firstname (field in customer Collection) CONTAINS 'Arnaud'