OpenCTI-Platform
Problems with elastic connector and in the output of data from Opencti
Prerequisites
- [x] I read the Deployment and Setup section of the OpenCTI documentation as well as the Troubleshooting page and didn't find anything relevant to my problem.
- [x] I went through old GitHub issues and couldn't find anything relevant
- [x] I googled the issue and didn't find anything relevant
Description
Hey guys
I'm setting up an elastic connector in openСTI, I see messages going from openСTI to elastic, but nothing is written to the elastic index. I’ve been racking my brain for almost two weeks now, there are no errors in the logs (I can provide them if necessary), I have a guess that everything is connected with this:
connector-elastic_1 | {"timestamp": "2023-10-02T09:47:33.159895Z", "level": "WARNING", "name": "elastic", "message": "For document id 39241d51-22f7-4d18-bfce-39f5f97ca807, entity is 'None'. Skipping."}
But I don't know how to fix it
Environment
- OS (where OpenCTI server runs): { Debian 12 }
- OpenCTI version: { e.g. OpenCTI 1.0.2 }
Reproducible Steps
Steps to create the smallest reproducible scenario:
Added elastic connector in OpenCTI's docker compose file Connector is fed config.yml in docker compose file I am using Elasticl self signed certs, and have linked them into the config.yml connector-elastic_1 | {"timestamp": "2023-10-03T05:48:19.874780Z", "level": "ERROR", "name": "pycti.api", "message": "('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))"} connector-elastic_1 | Traceback (most recent call last): connector-elastic_1 | File "/runtime/bin/elastic", line 8, in connector-elastic_1 | sys.exit(main()) connector-elastic_1 | ^^^^^^ connector-elastic_1 | File "/runtime/lib/python3.11/site-packages/elastic/console.py", line 225, in main connector-elastic_1 | ElasticInstance = ElasticConnector(config=config, datadir=datadir) connector-elastic_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ connector-elastic_1 | File "/runtime/lib/python3.11/site-packages/elastic/elastic.py", line 25, in init connector-elastic_1 | self.helper = OpenCTIConnectorHelper(config) connector-elastic_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ connector-elastic_1 | File "/runtime/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py", line 661, in init connector-elastic_1 | self.api = OpenCTIApiClient( connector-elastic_1 | ^^^^^^^^^^^^^^^^^ connector-elastic_1 | File "/runtime/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 217, in init connector-elastic_1 | raise ValueError( connector-elastic_1 | ValueError: OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
connector-elastic_1 | {"timestamp": "2023-10-02T07:30:46.917018Z", "level": "DEBUG", "name": "elastic", "message": "_process_message"} connector-elastic_1 | {"timestamp": "2023-10-02T07:30:46.917205Z", "level": "DEBUG", "name": "elastic", "message": "[PROCESS] Message (id: 1692173373074-0, date: 2023-08-16 08:09:33+00:00, data: {'id': 'external-reference--0d62c23a-209f-58d2-b20b-b9f02fc49f28', 'spec_version': '2.1', 'type': 'external-reference', 'extensions': {'extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba': {'extension_type': 'new-sdo', 'id': '0698c6ec-9c95-4344-8c33-25f914944738', 'type': 'External-Reference', 'created_at': '2023-08-16T08:09:33.074Z', 'updated_at': '2023-08-16T08:09:33.074Z', 'is_inferred': False, 'creator_ids': ['88ec0c6a-13ce-5e39-b486-354fe4a7084f']}}, 'source_name': 'NIST NVD', 'url': 'https://nvd.nist.gov/vuln/detail/CVE-2023-20564'})"}%22%7D) connector-elastic_1 | {"timestamp": "2023-10-02T07:30:46.917338Z", "level": "DEBUG", "name": "elastic", "message": "[CREATE] Processing indicator {external-reference--0d62c23a-209f-58d2-b20b-b9f02fc49f28}"} connector-elastic_1 | {"timestamp": "2023-10-02T07:30:46.930343Z", "level": "WARNING", "name": "elastic", "message": "For document id 0698c6ec-9c95-4344-8c33-25f914944738, entity is 'None'. Skipping."} worker_3 | File "/usr/local/lib/python3.11/http/client.py", line 1378, in getresponse connector-elastic_1 | {"timestamp": "2023-10-02T07:30:46.930588Z", "level": "DEBUG", "name": "elastic", "message": "_process_message"} connector-elastic_1 | {"timestamp": "2023-10-02T07:30:46.930808Z", "level": "DEBUG", "name": "elastic", "message": "[PROCESS] Message (id: 1692173374328-0, date: 2023-08-16 08:09:34+00:00, data: {'id': 'external-reference--c1f26a0f-3257-5e0e-8b16-cce4e07a5849', 'spec_version': '2.1', 'type': 'external-reference', 'extensions': {'extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba': {'extension_type': 'new-sdo', 'id': 'cc957c7b-f65e-446d-b685-b93c53281862', 'type': 'External-Reference', 'created_at': '2023-08-16T08:09:34.328Z', 'updated_at': '2023-08-16T08:09:34.328Z', 'is_inferred': False, 'creator_ids': ['88ec0c6a-13ce-5e39-b486-354fe4a7084f']}}, 'source_name': 'MISC', 'url': 'https://www.amd.com/en/corporate/product-security/bulletin/AMD-SB-7004'})"}%22%7D) connector-elastic_1 | {"timestamp": "2023-10-02T07:30:46.930981Z", "level": "DEBUG", "name": "elastic", "message": "[CREATE] Processing indicator {external-reference--c1f26a0f-3257-5e0e-8b16-cce4e07a5849}"}
@TIexplorer do you still have an issue with this? Additionally, feel also free to post your question on slack to get a wider visibility on your issue!
@nino-filigran
@TIexplorer do you still have an issue with this? Additionally, feel also free to post your question on slack to get a wider visibility on your issue!
Hi, The problem persists, unfortunately Slack did not give me an answer to this question
@Kedae Are you able to help here, based on the data provided? @TIexplorer can you precise the version of your platform?
connector-elastic_1 | ValueError: OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
This message seems to indicate that you have a bad configuration on your connector. Can you check or share your configuration ?
Closing this ticket for now given there's no activity since a while. Feel free to re-open it if you have any questions.