Problems with MISP connector

[mahmut-the-guy]

Prerequisites

• [x] I read the Deployment and Setup section of the OpenCTI documentation as well as the Troubleshooting page and didn't find anything relevant to my problem.
• [x] I went through old GitHub issues and couldn't find anything relevant
• [x] I googled the issue and didn't find anything relevant

Description

1. I had a MISP connector running for a week, it ended up accumulating 35 million messages, workers acknowledging almost none, with bundles per second nearing zero at all times.

   Yesterday, I purged all messages and reset the connector state. Initially, some entities started to be ingested but buildup occurred eventually, now sitting at 4m bundles. MISP connector especially seems to be problematic when running from scratch.

   For reference, I'm running 4 workers with one platform node. Throwing more workers does not seem to help at all in general. (which is odd, interested to know the why)

2. In another instance, the platform also suffers from slow ingestion and the following error is repeatedly reported via frontend:

   {"name": "MISSING_REFERENCE_ERROR", "message": "Element(s) not found"}
   

   What does this error mean specifically? How could we fill in these missing references?

Environment

1. OS (where OpenCTI server runs): Ubuntu 22.04.2 LTS on Docker Compose
2. OpenCTI version: { e.g. OpenCTI 5.12.24 }

Reproducible Steps

I believe using the Docker configuration provided by OpenCTI, and enabling MISP connector will eventually result in this problem i.e., slow and (eventually stalling) data entry.