API crash on stress test

[mordebip]

When building from master and running a restfull stress test, the api crashes at less than 1k queries (server using an 8700k and 64gb of ram)

i'm targeting a modified version of the post query of the chat widget (modified to return the chat response directly)

(it happens when i set the response at less than 10ms).

PS : debug execution

Browserslist: caniuse-lite is outdated. Please run next command yarn upgrade caniuse-lite browserslist FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory <--- Last few GCs ---> [69:0x38575c0] 46350 ms: Mark-sweep 1389.4 (1424.6) -> 1389.4 (1424.6) MB, 204.5 / 0.0 ms (average mu = 0.072, current mu = 0.000) last resort GC in old space requested [69:0x38575c0] 46570 ms: Mark-sweep 1389.4 (1424.6) -> 1389.4 (1424.6) MB, 220.7 / 0.0 ms (average mu = 0.036, current mu = 0.000) last resort GC in old space requested <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x222bd5d9e6e9 <JSObject> 0: builtin exit frame: parse(this=0x222bd5d91a21 <Object map = 0x221ba70842a9>,0x2ccf175f2441 <Very long string[10921608]>,0x222bd5d91a21 <Object map = 0x221ba70842a9>) 1: deserialize [0x33a1ec84c759] [/usr/src/app/node_modules/elasticsearch/src/lib/serializers/json.js:~42] [pc=0x3206ef65526a](this=0x210a39748139 <Json map = 0x2ec8ff7b309>,str=0x2ccf175f2441 <Very long string[10921608]>... 1: 0x8f9d10 node::Abort() [/usr/local/bin/node] 2: 0x8f9d5c [/usr/local/bin/node] 3: 0xaffd0e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node] 4: 0xafff44 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node] 5: 0xef4152 [/usr/local/bin/node] 6: 0xf0396f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node] 7: 0xecbbb5 [/usr/local/bin/node] 8: 0xed342a v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node] 9: 0x101ce9d v8::internal::String::SlowFlatten(v8::internal::Handlev8::internal::ConsString, v8::internal::PretenureFlag) [/usr/local/bin/node] 10: 0xbd4569 v8::internal::Builtin_JsonParse(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node] 11: 0x3206eed5bf7d [nodemon] files triggering change check: core [nodemon] changes after filters (before/after): 1/0 [nodemon] files triggering change check: core [nodemon] changes after filters (before/after): 1/0 [nodemon] files triggering change check: core [nodemon] changes after filters (before/after): 1/0 [nodemon] files triggering change check: core [nodemon] changes after filters (before/after): 1/0 Aborted (core dumped) [nodemon] app crashed - waiting for file changes before starting...

[mordebip]

adding --max-old-space-size=16000 to the package.json file solved the issue, i was able to make jmeter tests with 1000 threads.

[wrathagom]

Can you provide information on the version of Articulate you're running, the command you're using to execute these tests?

[wrathagom]

@mordebip please see above.

[bbm-design]

i tried it using the latest master branch that i built.

Concerning the tests, i updated the file \api\lib\channels\chat-widget\post.js

import Boom from 'boom'; module.exports = async function ({ connection, request, h }) { const { agentService, channelService } = request.services(); const event = request.payload; event.server = request.server; const sessionId = event.sessionId; try { var response = await agentService.converse({ id: connection.agent, sessionId, text: event.text, timezone: event.timezone ? event.timezone : null, debug: false, additionalKeys: { ubiquity: { connection, event } } }); return response.processedResponses[0].textResponse; } catch ({ message, statusCode }) { return new Boom(message, { statusCode }); } };

and then i send post queries on http://srv/connection/id/external/ by providing the text and sessionid on the post body

[wrathagom]

you still haven't said what you're using to make the stress test calls, how many requests you consider to be a stress test, etc.

[bbm-design]

i said that earlier,

i used jmeter with 1000 threads for 60 seconds, querying the same message on the same sesssion.

The idea was to simulate 1000 simultaneous queries at the same time.

[mordebip]

i did the same thing as @Blue-Beam-com said, the crash happened even with restfullstress chrome app

[dcalvom]

I'm passing trough all issues. We need to work on this, @mordebip now that you have been working more with articulate let us know if you have any other comment.

[mordebip]

I wasn't really able to track the memory issue as i don't really know how stress tests works and how to troubleshoot (i'm pretty unfamiliar with js too)

I have to update the version and recheck, the webapp is fairly stable, i should retest the connections (webchat/facebook/...) with the authentication enabled, this part was not clear for me.