client-python icon indicating copy to clipboard operation
client-python copied to clipboard

Handle Connection hangups properly

Open MaxwellDPS opened this issue 2 years ago • 0 comments

Description

Any hiccup to the graphql API will cause an API error, and makes calling .list() at scale impractical as it cant hold page / state.

This error handling is in-scope of the SDK

Environment

  1. OS (where OpenCTI server runs): CentOS Stream 9
  2. OpenCTI version: 10.5.2
  3. Other environment details: Scaled Cluster Deployment

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Use the SDK.

Expected Output

Automatically handle retries and connection hangups rather then raise an exception if a retry works

Actual Output

{"timestamp": "2023-09-26T17:04:31.821244Z", "level": "INFO", "name": "pycti.entities", "message": "Listing StixCyberObservables after <REMOVED>"}
5
{"timestamp": "2023-09-26T17:04:32.182713Z", "level": "INFO", "name": "pycti.entities", "message": "Listing StixCyberObservables after <REMOVED>"}
4
{"timestamp": "2023-09-26T17:04:32.325378Z", "level": "INFO", "name": "pycti.api", "message": "upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111"}
3
{"timestamp": "2023-09-26T17:04:32.398060Z", "level": "ERROR", "name": "root", "message": "Error in message processing, reporting error to API", "exc_info": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py\", line 235, in _process_message\n    json_data[\"internal\"][\"work_id\"], message_task.result()\n                                      ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/opencti-connector-export-file-csv/export-file-csv.py\", line 333, in _process_message\n    entities_list = do_list(\n                    ^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py\", line 401, in list\n    result = self.opencti.query(\n             ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 365, in query\n    raise ValueError(r.text)\nValueError: upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111"}
2
{"timestamp": "2023-09-26T17:04:32.398405Z", "level": "INFO", "name": "pycti.api", "message": "Reporting work update_processed work_622e884a-b3fe-401d-8426-6b27e12f283e_2023-09-26T17:03:46.665Z"}

Additional information

Present since release

MaxwellDPS avatar Sep 26 '23 17:09 MaxwellDPS