opencti
opencti copied to clipboard
Published
20 hours ago •
OpenCTI-Platform
OpenCTI-Platform
[6.1.X] Platform Crashing - JavaScript heap out of memory
Description
JavaScript heap out of memory OOMs plaguing uptime and usability of 6.1.X - Seems linked to import of data
Environment
- OS (where OpenCTI server runs): CentOS Stream 9 - Kubernetes
- OpenCTI version: 6.1.6
- OpenCTI client: Frontend
- Other environment details: scaled, clustered deployment
Reproducible Steps
Steps to create the smallest reproducible scenario:
None known - Has been continuous since the 6.1.X upgrade Seems to be far worse when data is being imported.
RabbitMQ is showing a backlog of ~1k messages in the push_sync queue that are going nowhere. (Connector queues are dropping)
Redis memory is spiking at the times this is happening
Expected Output
no platform crashes
NAME READY STATUS RESTARTS AGE
opencti-opencti-api-f84c7f588-94rsq 2/2 Running 0 15m
opencti-opencti-api-f84c7f588-hq5zx 2/2 Running 0 15m
opencti-opencti-api-f84c7f588-lf25m 2/2 Running 0 15m
opencti-opencti-api-f84c7f588-scb4v 2/2 Running 0 15m
opencti-opencti-api-f84c7f588-zxgbq 2/2 Running 0 15m
...
opencti-opencti-web-6d5656fc4f-7jg9p 2/2 Running 0 15m
opencti-opencti-web-6d5656fc4f-7lng5 2/2 Running 0 15m
opencti-opencti-web-6d5656fc4f-fvllc 2/2 Running 0 15m
opencti-opencti-web-6d5656fc4f-qjsgr 2/2 Running 0 15m
Actual Output
NAME READY STATUS RESTARTS AGE
opencti-opencti-api-f84c7f588-94rsq 2/2 Running 2 (5m20s ago) 15m
opencti-opencti-api-f84c7f588-hq5zx 2/2 Running 2 (6m25s ago) 15m
opencti-opencti-api-f84c7f588-lf25m 2/2 Running 1 (118s ago) 15m
opencti-opencti-api-f84c7f588-scb4v 2/2 Running 0 15m
opencti-opencti-api-f84c7f588-zxgbq 2/2 Running 1 (4m26s ago) 15m
...
opencti-opencti-web-6d5656fc4f-7jg9p 2/2 Running 2 (7m23s ago) 14m
opencti-opencti-web-6d5656fc4f-7lng5 2/2 Running 1 (3m2s ago) 15m
opencti-opencti-web-6d5656fc4f-fvllc 2/2 Running 0 7m56s
opencti-opencti-web-6d5656fc4f-qjsgr 2/2 Running 1 (66s ago) 15m
Additional information
Logs start up to crash
{"category":"APP","environment":"production","level":"info","message":"[OPENCTI] Starting platform","source":"backend","timestamp":"2024-06-20T20:17:33.483Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI] Checking dependencies statuses","source":"backend","timestamp":"2024-06-20T20:17:33.485Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[SEARCH] Engine client not specified, trying to discover it with opensearch client","source":"backend","timestamp":"2024-06-20T20:17:33.494Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[SEARCH] Engine detected to elk","source":"backend","timestamp":"2024-06-20T20:17:33.568Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[SEARCH] elk (8.13.4) client selected / runtime sorting enabled / attachment processor enabled","source":"backend","timestamp":"2024-06-20T20:17:33.604Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[CHECK] Search engine is alive","source":"backend","timestamp":"2024-06-20T20:17:33.604Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[CHECK] File engine is alive","source":"backend","timestamp":"2024-06-20T20:17:33.637Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[CHECK] RabbitMQ engine is alive","source":"backend","timestamp":"2024-06-20T20:17:33.704Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[REDIS] Redis 'base' client ready","source":"backend","timestamp":"2024-06-20T20:17:33.732Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[REDIS] Clients initialized in single mode","source":"backend","timestamp":"2024-06-20T20:17:33.733Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[CHECK] Redis engine is alive","source":"backend","timestamp":"2024-06-20T20:17:33.733Z","version":"6.1.6"}
{"category":"APP","level":"warn","message":"SMTP seems down, email notification will may not work","source":"backend","timestamp":"2024-06-20T20:17:38.816Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[CHECK] Python3 is available","source":"backend","timestamp":"2024-06-20T20:17:38.846Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[REDIS] Redis 'subscriber' client ready","source":"backend","timestamp":"2024-06-20T20:17:38.854Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Cache manager pub sub listener initialized","source":"backend","timestamp":"2024-06-20T20:17:38.855Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[REDIS] Redis 'lock' client ready","source":"backend","timestamp":"2024-06-20T20:17:38.864Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[INIT] Starting platform initialization","source":"backend","timestamp":"2024-06-20T20:17:39.833Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[INIT] Existing platform detected, initialization...","source":"backend","timestamp":"2024-06-20T20:17:39.883Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[INIT] admin user initialized","source":"backend","timestamp":"2024-06-20T20:17:44.147Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[MIGRATION] Read 15 migrations from the database","source":"backend","timestamp":"2024-06-20T20:17:44.215Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[MIGRATION] Platform already up to date, nothing to migrate","source":"backend","timestamp":"2024-06-20T20:17:44.219Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[MIGRATION] Migration process completed","source":"backend","timestamp":"2024-06-20T20:17:44.219Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[MIGRATION] Platform version updated to 6.1.6","source":"backend","timestamp":"2024-06-20T20:17:44.254Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[INIT] Platform initialization done","source":"backend","timestamp":"2024-06-20T20:17:44.302Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI] API ready on port 8080","source":"backend","timestamp":"2024-06-20T20:17:45.398Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Expiration manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.398Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Connector manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.398Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Starting Import Csv built in connector manager","source":"backend","timestamp":"2024-06-20T20:17:45.398Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Retention manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Task manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Rule engine not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Sync manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Ingestion manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] History manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Notification manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Publisher manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Playbook manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Starting file index manager","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Indicator decay manager not started (disabled by configuration)","source":"backend","timestamp":"2024-06-20T20:17:45.473Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Starting Garbage collection manager","source":"backend","timestamp":"2024-06-20T20:17:45.474Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Starting Telemetry manager","source":"backend","timestamp":"2024-06-20T20:17:45.474Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Starting cluster manager","source":"backend","timestamp":"2024-06-20T20:17:45.474Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Support Package pub sub listener initialized","source":"backend","timestamp":"2024-06-20T20:17:45.491Z","version":"6.1.6"}
{"category":"APP","connectorId":"d336676c-4ee5-4257-96ff-b2a86688d4af","level":"info","message":"[QUEUEING] Starting connector queue consuming","source":"backend","timestamp":"2024-06-20T20:17:45.529Z","version":"6.1.6"}
{"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500,"referer":"https://<rem>/dashboard/threats/intrusion_sets/610ea783-fafb-49a3-b41f-3ab14d11cd35"},"message":"Http call interceptor fail","name":"UNKNOWN_ERROR","stack":"UNKNOWN_ERROR: Http call interceptor fail\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at UnknownError (/opt/opencti/build/src/config/errors.js:82:47)\n at fn (/opt/opencti/build/src/http/httpPlatform.js:455:18)\n at lle.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:71:5)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n at lle.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:67:12)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n at lle.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:67:12)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n at lle.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:67:12)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)"},{"message":"stream is not readable","name":"InternalServerError","stack":"InternalServerError: stream is not readable\n at readStream (/opt/opencti/build/node_modules/raw-body/index.js:185:17)\n at getBody (/opt/opencti/build/node_modules/raw-body/index.js:116:12)\n at read (/opt/opencti/build/node_modules/body-parser/lib/read.js:79:3)\n at fn (/opt/opencti/build/node_modules/body-parser/lib/types/json.js:138:5)\n at lle.handle [as handle_request] (/opt/opencti/build/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:328:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n at cors (/opt/opencti/build/node_modules/cors/lib/index.js:188:7)\n at cb (/opt/opencti/build/node_modules/cors/lib/index.js:224:17)\n at originCallback (/opt/opencti/build/node_modules/cors/lib/index.js:214:15)\n at cb (/opt/opencti/build/node_modules/cors/lib/index.js:219:13)\n at optionsCallback (/opt/opencti/build/node_modules/cors/lib/index.js:199:9)\n at fn (/opt/opencti/build/node_modules/cors/lib/index.js:204:7)\n at lle.handle [as handle_request] (/opt/opencti/build/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:328:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n at Function.handle (/opt/opencti/build/node_modules/express/lib/router/index.js:175:3)\n at router (/opt/opencti/build/node_modules/express/lib/router/index.js:47:12)"}],"level":"error","message":"Http call interceptor fail","source":"backend","timestamp":"2024-06-20T20:20:27.622Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[REDIS] Redis 'publisher' client ready","source":"backend","timestamp":"2024-06-20T20:22:16.517Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[FILE STORAGE] delete file import/global/PRC Malware Infrastructure 1-21 March.pdf in index","source":"backend","timestamp":"2024-06-20T20:23:25.040Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[TELEMETRY] File exporter activated","source":"backend","timestamp":"2024-06-20T20:28:46.210Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[TELEMETRY] Otlp exporter activated","source":"backend","timestamp":"2024-06-20T20:28:47.031Z","version":"6.1.6"}
{"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running Telemetry manager infinite cron handler","source":"backend","timestamp":"2024-06-20T20:28:47.049Z","version":"6.1.6"}
<--- Last few GCs --->
[7:0x7f21a05eb690] 744891 ms: Mark-Compact 8445.5 (8557.9) -> 8445.4 (8557.1) MB, 3235.78 / 0.00 ms (average mu = 0.465, current mu = 0.093) allocation failure; scavenge might not succeed
[7:0x7f21a05eb690] 749751 ms: Mark-Compact 8459.8 (8558.8) -> 8459.8 (8587.6) MB, 4840.97 / 0.00 ms (average mu = 0.242, current mu = 0.004) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
