Uncaught RedisClient error: RangeError: Invalid string length

[rlynch-ironnet]

Description

I am seeing an uncaught error, crashing the platform/container. It is generally seen after a handled quorum error from the redis client.

{
  "category": "APP",     
  "error": {         
    "context": {},
         "message": "The operation was unable to achieve a quorum during its retry window.",
         "name": "ExecutionError",
         "stack": "
                ExecutionError: The operation was unable to achieve a quorum during its retry window.\n
                at iL._execute (/opt/opencti/build/node_modules/redlock/src/index.ts:462:15)\n
                at runMicrotasks (<anonymous>)\n
                at processTicksAndRejections (node:internal/process/task_queues:96:5)\n
                at Object.unlock (/opt/opencti/build/src/database/redis.ts:308:11)\n
                at createEntityRaw (/opt/opencti/build/src/database/middleware.js:2919:15)\n
                at createEntity (/opt/opencti/build/src/database/middleware.js:2939:16)\n
                at addReport (/opt/opencti/build/src/domain/report.js:131:19)"
     },
     "level": "warn",
     "locks": [
         "report--7a71f375-6f0d-59b6-b2d3-768eda3bc48c",
         "report--0183585f-3dc1-5664-b725-f764465e8d41"
     ],
     "message": "[REDIS] Failed to unlock resource",
     "timestamp": "2022-09-20T21:09:58.039Z",
     "version": "5.3.13"
 }

--
RangeError: Invalid string length
at JSON.stringify (<anonymous>)
at t.PubSubAsyncIterator (/opt/opencti/build/node_modules/graphql-redis-subscriptions/src/redis-pubsub.ts:87:98)
at PubSubAsyncIterator (/opt/opencti/build/node_modules/graphql-redis-subscriptions/src/pubsub-async-iterator.ts:33:14)
at Object.PubSubAsyncIterator [as next] (/opt/opencti/build/node_modules/graphql-redis-subscriptions/src/pubsub-async-iterator.ts:33:14)
at PubSubAsyncIterator (/opt/opencti/build/node_modules/graphql-redis-subscriptions/src/pubsub-async-iterator.ts:33:14)
at new Promise (<anonymous>)
at vSr (/opt/opencti/build/node_modules/graphql-redis-subscriptions/src/pubsub-async-iterator.ts:33:14)
at t.publish (/opt/opencti/build/node_modules/graphql-redis-subscriptions/src/redis-pubsub.ts:86:54)
at notify (/opt/opencti/build/src/database/redis.ts:151:12)
at addReport (/opt/opencti/build/src/domain/report.js:132:10)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)

Environment

1. OS - AWS Fargate (2vcpu, 4gb mem)
2. OpenCTI version: 5.3.13
3. OpenCTI client: backend
4. Other environment details:

• Single host for web UI
• up to 3 platform containers for up to 15 workers during autoscaling

Reproducible Steps

Steps to create the smallest reproducible scenario: I have been unable to recreate this issue.

Expected Output

At a minimum, handle the error, then shutdown the container so it is printed with structured logging. Best case: find a root cause.

Actual Output

See stacktrace above.

Additional information

n/a

Screenshots (optional)

n/a

[SamuelHassine]

Hello @rlynch-ironnet,

Are you using a Redis cluster (OpenCTI is not yet compatible with Redis cluster). Do you have the Redis logs?

Kind regards, Samuel

[rlynch-ironnet]

Single node, with the following extra startup flags. { name = "REDIS_EXTRA_FLAGS", value = "--maxmemory ${floor(var.redis_memory_size * 0.80)}mb --maxmemory-policy allkeys-lru" },

Redis data is mounted to an EFS network drive (thanks fargate) via { containerPath = "/data", sourceVolume = "redis_efs" }

Here's some logs from around the time of the crash log above. https://gist.github.com/rlynch-ironnet/7d8abc10d8b9915162d70b97f3dcc0b4

[rlynch-ironnet]

From what I've seen, the container's resources are typically not very stressed.

opencti_redis_trimming                 = 50000
redis_image = "bitnami/redis"
redis_tag         = "7.0.4"
redis_cpu_size    = 512
redis_memory_size = 2048