connectors
connectors copied to clipboard
OpenCTI-Platform
elasticsearch connector unable process message
Description
When using the connector, I encountered the error that I will share below, and the connector was terminated I think it's because of a bunch of iocs that start with http, but I'm not sure if that's the reason
Can anyone help me? I want to send iocs from opencti to elasticsearch
Environment
- OS (where OpenCTI server runs):ubuntu 22
- OpenCTI version: 6.0.7
- elastic connector version: 6.0.7
elastic connector config ======> connector-elastic: image: opencti/connector-elastic:6.0.7 environment: - OPENCTI_URL=http://myopencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} - CONNECTOR_ID=${CONNECTOR_ELASTIC_ID} - CONNECTOR_LIVE_STREAM_ID=1a452269-43ff-46ec-9ffd-4c472613414a - CONNECTOR_LIVE_STREAM_LISTEN_DELETE=true - CONNECTOR_LIVE_STREAM_NO_DEPENDENCIES=true - "CONNECTOR_NAME=OpenCTI Elastic Connector" - CONNECTOR_SCOPE=elastic - CONNECTOR_CONFIDENCE_LEVEL=80 # From 0 (Unknown) to 100 (Fully trusted) - CONNECTOR_LOG_LEVEL=error - "CONNECTOR_ENTITY_NAME=Elastic Detection Cluster" - "CONNECTOR_ENTITY_DESCRIPTION=Elasticsearch detection engine cluster" - CONNECTOR_MODE=ecs # Options # - ELASTICSEARCH_APIKEY= - ELASTICSEARCH_HOSTS=http://elasticsearchip:9200 - ELASTICSEARCH_USERNAME=username - ELASTICSEARCH_PASSWORD=password # - ELASTICSEARCH_SSL_VERIFY=False restart: always depends_on: - opencti
error log ======>
[2024-03-28T10:49:35.104][WARNING] Config file 'config.yml' does not exist. Relying on environment and defaults.
ERROR ERROR ERROR ERROR Unable to process the message: {"version":"4","type":"create","scope":"external","message":"creates a Url http:// | ERROR ERROR ERROR ERROR Unable to process the message: {"version":"4","type":"create","scope":"external","message":"creates a Url http:// | timestamp=2024-03-28T10:49:35.769468Z ERROR ERROR ERROR ERROR Unable to process the message: {"version":"4","type":"create","scope":"external","message":"creates a Url http:// | ERROR ERROR ERROR ERROR Unable to process the message: {"version":"4","type":"create","scope":"external","message":"creates a Url http:// | timestamp=2024-03-28T10:49:35.769468Z name=elastic
ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | timestamp=2024-03-28T10:49:35.777905Z ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | timestamp=2024-03-28T10:49:35.777905Z name=OpenCTI Elastic Connector ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | timestamp=2024-03-28T10:49:35.777905Z ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | timestamp=2024-03-28T10:49:35.777905Z name=OpenCTI Elastic Connector exc_info=Traceback (most recent call last):
File "/opt/opencti-connector-elastic/elastic/elastic.py", line 156, in _process_message
data = json.loads(msg.data)["data"]
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/init.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 61 (char 60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py", line 633, in run self.callback(msg) File "/opt/opencti-connector-elastic/elastic/elastic.py", line 159, in _process_message raise ValueError("Cannot process the message: " + msg) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~ TypeError: can only concatenate str (not "Event") to str ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | timestamp=2024-03-28T10:49:35.777905Z ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | timestamp=2024-03-28T10:49:35.777905Z name=OpenCTI Elastic Connector ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | timestamp=2024-03-28T10:49:35.777905Z ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | ERROR ERROR ERROR ERROR Error in ListenStream loop, exit. | timestamp=2024-03-28T10:49:35.777905Z name=OpenCTI Elastic Connector exc_info=Traceback (most recent call last): File "/opt/opencti-connector-elastic/elastic/elastic.py", line 156, in _process_message data = json.loads(msg.data)["data"] ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/init.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) ^^^^^^^^^^^^^^^^^^^^^^ json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 61 (char 60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py", line 633, in run self.callback(msg) File "/opt/opencti-connector-elastic/elastic/elastic.py", line 159, in _process_message raise ValueError("Cannot process the message: " + msg) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~ TypeError: can only concatenate str (not "Event") to str attributes={"reason":"can only concatenate str (not "Event") to str"} Terminated
